there's also probably load balancers that downgrade models during high use.

at this point frontier companies are just selling upgraded harnesses and tool calls with the rest of us

The good part is showing that the drop in thinking tokens persists no matter what grouping you slice across. They make a very persuasive case that there's something systematic going on.

My usual complaint about these "they're nerfing the models, I feel it in my bones!" posts is that they don't account for the workload changing. From working on my own stuff, there are a series of evolutionary/de-evolutionary changes that happen in a heavily AI-written codebase. Initially everything goes great. Then the AI takes on too much technical debt. Improvements slow down and regressions creep up until it becomes a never-ending game of whack-a-mole just to keep up. So you direct some (probably AI) effort towards cleaning things up, reducing duplication, and removing patches for problems that are better fixed with a design change, or workarounds because the harness saw the wrong version or you incorrectly described a problem and it strenuously solved a non-problem. That gets you back up to cruising speed for a while, then the project exceeds some hidden threshold for size in latent space or something, and further progress has to rely on attending to one aspect of the codebase at a time. Once again, the architecture becomes the limiting factor, but in a subtly different way. My sense is that it all boils down to some sort of "attention capacity" -- is your codebase and problem space amenable to looking at one aspect at a time, or is it all snarled together? -- but that's an essay that I'd love to write but really don't have enough experience to do justice.

Anyway, the details don't matter. The point is that not only can you not assume that the difficulty presented to the AI is roughly constant over time, but also there's evidence to believe that it will be normally be increasing. (Unless you're constantly starting new projects instead of continuing old ones.)

That's why I like this writeup. Focusing on thinking trails doesn't eliminate the problem of snowballing difficulty, but it does sidestep the worst of it. In fact, I'd expect the same setup to think more as the complexity/sloppiness creeps up.

The flawed part that bothered me was that it feels like there's a little bit of a predetermined conclusion that thinking is a magic sauce that makes everything taste better if you spread it on everything. I want a high variance on thinking, especially between interactions. A smarter model would have a higher variance, in my opinion. So the accusatory tone (perhaps I should reread it? My first impressions are often wrong) around "look! it doesn't bother to think at all a lot of the time. That can't be right!" seems misguided to me. It should think when it needs to, and if its thinking was clear then it won't need to re-think over and over again; it's all still in the context.

Forgive the anthropomorphization, but consider those studies of chess experts vs novices. Novices have to work way harder, working through all kinds of things from scratch, while the expert instantly and effortlessly recognizes what's going on.

But anyway, the main takeaway fully survives this criticism. The models appear to systematically think less over time. It doesn't matter if a smarter model might be able to think less for the same quality; this is happening over the same model.

(ie, even a pause in AI training isn't going to stop the train where AI flips the economy upside down, we've barely even seen the impact of the current frontier)

This cycle of new model running at full quantisation and then nerfed few days / weeks after premiere should be called out. Anthropic should also drop the adaptive reasoning scam.

If I pay for Fable, I should get full, not nerfed model at honest pricing.

Regulators should investigate them.

OpenAI is no different. Astra has basically the same problem.

I'm thinking they're all running out of physical resources. It's the DotCom bubble all over again; rollout of the physical infrastructure that's necessary to keep all of the pie-in-the-sky promises will not happen on the timescales that investors can work with, and they will panic when they realize this.

EDIT: And, frankly, I can't wait. I'm tired of the sketchy and dishonest way these companies are behaving.

I was rooting for Anthropic 2 years ago, but now I have become an extremely bitter customer. Just another version of OpenAI, if not shittier.

I'm getting really sick of the constant rot and "magic breakthrough" cycle, so im going full local, at expense on paper but being able to trust something which I need to understand the reliability of is priceless.

I like predictable. I'll take slightly less capable over unreliably capable, since with reliable i can calibrate my expectations and learn what aspects of my workflows to entrust and trust it will work. You simply cannot do that with models you don't control and in my experience they will all errode after the initual marketing wave passes, likely you eventually get fed heavily quantized versions and are expected to accept degraded service when what convinced you to pay was a far superior product. No such issues with local.

Even Altman called out Anthropic? Isn't Anthropic the biggest competitor Sam Altman has?

Where?

I see so many accusations of this happening and it's so easy to check, but nobody ever proves it.

I imagine some people have their own personal in depth benchmarks they could do this for.

Oh, yes there is. DeepSeek 4.1 Flash on max thinking can simply be dropped into Claude Code. Close your eyes as the chain-of-thought traffic scrolls by and you can easily fool yourself into thinking you're still running Opus, in terms of both cognition and throughput.

To be fair, matching Opus's throughput costs about as much as a new car, but cars suck nowadays and you didn't want a new one anyway, right...? Failing that, rent a cloud server, one that you control.

They might but multiple competitors engaging in ongoing deception as an intentional corporate strategy isn't required to explain what we're seeing. It's entirely possible to get the same clearly unethical outcome without any employees knowingly participating in an explicitly unethical plan of record.

Instead it happens without overt coordination when individuals and groups within an org each pursue their local metrics and incentives. In isolation, no individual action seems obviously unethical on its own. They just look like 'optimizing performance', 'maintaining ASP or ARPU targets' or 'achieving operating margin', etc. Customers are still getting deceived and receiving less for their money than they think. The difference is most of the people involved in enabling it get to not feel bad about themselves.

Sam Altman believes he can train a model entirely on synthetic data, which he admits would not have human world knowledge but is interesting none the less, which likely led to their mathematical models.

Overall models have become cheaper to run and smarter per token.

https://www.tomshardware.com/features/crucial-p2-ssd-qlc-fla...

https://www.tomshardware.com/news/wd-blue-sn550-ssd-performa...

https://www.tomshardware.com/news/adata-switches-nand-on-sx8...

The best case scenario is a situation like Y2K: a ton of people coordinate and work hard to produce no perceptible change, because unlike catastrophe, averting catastrophe feels boring.

Absolutely none of this points to “stagnant.” Stagnant is a terrible description of the AI industry.

A very small raise in prices may cause a very large loss in customers that you risk never getting back.

For example if customers figure out that the Chinese models are just as good, they are gone because they are so much cheaper.

Unlike you, however...

Of course the closed Anthropic would never share this, it would definitely take away the 'magic' feeling of the AI

I have quite strongly told them, in no uncertain terms, that they are going to kill themselves doing that.

If this is duress of competition or at gunpoint of regulators is up for the population to decide.

I find it odd, because it's not just plausible, it's understandable that they would need to dynamically throttle these things or degrade people to a different model. I would be fine with that if there was transparency. I'd think a business building any type of logic on AI decision making would like to know if they are actually using Opus and not Fable. But apparently the idea that these companies don't have infinite resources is conspiratorial fear mongering to most AI boosters.

However, I believe that runtime model quantization is possible with some publicly-available inference engines (e.g. vLLM), so its not beyond belief that the closed labs do quantize at runtime, either to allocate compute, or to nudge users towards a preferred model (e.g. make the incumbent model dumber to push people to use the latest-and-greatest model, or vice versa to ease the load on the latest model, which is typically larger than the old one).

They can limit how hard the model thinks for a given effort. Suddenly xhigh only thinks as hard as high did, and high shifts down to medium effort, and so on.

They can also serve quantized models. And this has the benefit of practically not showing up in benchmarks at all even if the user experience is obviously degraded.

The other major thing the labs do is silently drop the usage limits. This has become very noticeable for codex users who are suddenly burning through their weekly usage in a few hours.

I have no reason to doubt the claims of the employees at OpenAI and Anthropic who have told us personally multiple times, including here on HN, that they do not degrade the models in order to reduce load.

As for the endlessly long analysis in the OP, it appears it's based on analyzing their random usage data rather than any fixed benchmark. I don't think it makes much sense.

I think that’s the claim in the post, that even though no one can see the true chain of thought, that even the “thinking” text that does get exposed to the user is shorter given the same prompts over time. Not saying it’s true but I think that’s the claim. I’ve personally never noticed the alleged “nerfing” with my enterprise use at work or my subscription use at home which is only during off hours.

However, I also agree that I haven't seen any robust data from someone tracking it daily/weekly. The handful of sites purporting to do this aren't even running it enough times to hit stat sig.

edit: someone linked one elsewhere in this thread called AI Stupid Level - they "run 7 trials instead of just 1". I don't blame them. Doing this in a statistically sound manner would cost a small fortune.

As far as the API goes, it would be really obvious. I run a small service that uses LLMs extensively, and if a model suddenly dropped in performance it would be straightforward for us to prove it. We regularly run comparisons where we generate completions with alternative models to e.g. see if we could get away with using cheap models for easy cases, if the baseline outputs deteriorated it would be all over our metrics.

...I mean, if they were actually doing this despite saying that they don't—promising one product and delivering something else—I think that would be fraud, no?

And, maybe it's one thing to secretly defraud normies like us (although class action lawsuits do exist), but I don't think major enterprises or the US military would take too kindly to it.

> What is actually stopping these model companies

You can say this about any company in the world, selling anything.

It's trivially measurable, and there are people running the same benchmark on the leading models every day and measuring if they degrade. Spoiler: they don't.

But you can always say "the conspiracy goes higher", and that the companies know about these daily benchmarks and are routing them to "quality" envs.

Do you know why nobody outside the companies knows what's going on? Because they sell a black box with magic inside while steadfastly refusing to tell you if they are pushing buttons on said box while it is running.

Can you imagine how much fraud would exist in the gambling industry if the gambling commission didn't exist at all? Everytime an industry is unregulated and has high costs of entry the entities in the industry abuse their customers. The incentives are much too high for them not to.

1/ if the point of setting xhigh or max effort on a "reasoning model" is -not- to have additional reasoning tokens applied to the problem, then I fear we are all using AI wrong 2/ the decorum with which you "review" someone's work in public is entirely your choice. the only "cheap trick" on the internet is being a pseudonymous ass.

Yeah, expecting the world when all you have is a 8GB graphics card? You're going to be disappointed.

16GB is table stakes (IQ3_XSS). 32 GB is better.

Perhaps reverse the question: Are your build/construct requirements just really counter-productive to how LLMs need to understand things?

I've found constructing the code, writing the tests, adding the docs; then running through them gets most of the way there.

I've also found that making a simple obvious edit is a useless endevour when the LLM is primed for the long context tasks.

So, again, the question is reversed: are you over reliant on the LLM to do even stupid simple likes like editting a css variable?

The problem is they can't fit any frontier level open models.

Where'd you get 30 years from? Show your work.

Click the part at the end that says "View historical performance". They wait to collect more data about a new model before adding it to the overall charts.

The overall solution rate continues to climb when new models are considered.

> If you look at the graphs, the latest benchmarks are showing a pretty significant dip, and they match pretty well with some horrible experiences I've had in recent weeks. You can see token usage steadily going down, matching exactly what the author measured on his own.

The y-axis is amplified to make differences look larger than they are.

Hover over the dots to see the confidence interval. A 1-2% change means nothing.

Co-Authored By: Haiku 4.5

But here’s the standard question: At what speeds/other limiting factors?

The Anthropic post points to the latest fix on Sept 12, and the issues mentioned also only affected Sonnet 4 and Haiku. Opus was misbehaving just last week. You are choosing to not see the evidence of degration, 85% -> 75% is a generational dip in intelligence.