It's a combination of 1) inevitable diminishing returns -- we've done all the easy stuff and a lot of the hard stuff, and 2) institutional incentives now favor safe, predictable science, rather than high-risk speculative projects.

The paper being discussed in this article rings true to me but as I was pointing out on an earlier thread about it, I'm not sure that disruptive always has the meaning the authors intend.

I can think of papers or work in my field that would be relatively high in a disruption index in the way they define it, but that are probably specious. They're papers that spurred new lines of research that are flashy and sound trendy but are ultimately theoretically and empirically empty and problematic.

It worries me a bit because these types of indices tend to get latched onto, and then get misinterpreted and abused, in conformance with Goodhart's Law. Taken in context, the paper makes an important point, and adds a bit of important empirical information, but I think the situation might be worse or better than the article implies, because what's disruptive isn't necessarily good from the perspective of scientific progress.