I have codex right now purely because they gave me a month free of ChatGPT Pro, so I have been using it in between my usage resets with claude. Since it's "free money" for me I have been using it exclusively on xHigh.
One of my most frequent prompts is "hey codex worked on ____, but it didn't quite hit the mark, can we review the work..."
Yes, part of this is normal even within the same model -- you have the highest power model review the work for correctness, refactoring opportunities, and so on, but man I tell you, I don't know what it is about codex, this is obviously one guy's anecdote -- same prompting style, same repository documentation ala MD files, same skills, way different results.
All that to say, maybe the bug report is on to something here, and it can be fixed.
Not sure if I agree, but I do happen to use a fair bit of web harness as well, just because I find it to be much more effective at web search and a different type of reasoning. So I must agree a little or else I wouldn't do that.
- This tracker not showing any visible degradation. - Clearly incorrect answers being reported due to truncated thinking.
Is the tracker not measuring 'simpler' tasks that might get auto-sent to "low reasoning hell" even on high/xhigh? Is the clustering not actually causing reasoning misses in real-life coding, or not enough of a negative effect compared to the improvements made elsewhere? Something else?
Both the web harness and codex app/cli use "GPT-5.5".
Maybe some issue with adaptive thinking? Another point for local models I guess, don't have to worry about silent server side changes.
Edit: To follow up, it seems to happen quite often. Out of 10 runs of the exact same prompt, 4/10 had this 516 thinking token issue, and every one of these had the wrong solution. So nearly half the time, 5.5 xhigh could be short circuiting and degrading performance. Granted the sample size is small.
Why there is so much effort in making adaptive thinking happen and don’t we train models to produce the end of thinning token better?
Feels like a bandaid. We need models to be trained to do a reasonable amount of reasoning (no pub intended):
reason
estimate remaining uncertainty
continue?
reason more
repeatSequential-thinking was really a step in the right direction, and works almost exactly how you've described, though when it was popular before the reasoning models and even now when I tried it recently I have never once see it use its branching feature and it tends also to have the RLHF urge to answer something "helpful" quickly instead.
But the larger problem is sound, and the answer is something jointly optimized (idk how they do the routing) but it’s hard to shoehorn it into the current paradigm.
I wonder if it's somehow getting confused between what's supposed to be an intermediate update vs the final result.
[1] https://github.com/openai/codex/issues/30364#issuecomment-48...
With OpenAI, at least my gpt-5.5 is the same as your gpt-5.5. You can't say that about glm for example.
My journey dealing with this has been transitioning from 5.5 high to 5.5 xhigh to 5.4 high.
5.4 high has been perfectly reliable for me for the last 3 weeks, and I am happy there.
Occasionally, I run some tasks on 5.5 xhigh to check if it has gone back to being 100% perfectly reliable, but, at this point, I am assuming they are just counting on releasing 5.6 rather than dealing with this reliability issue.
import os, glob, re
import matplotlib.pyplot as plt
vals = []
for f in glob.glob(os.path.expanduser(r"~\.codex") + r"\**\*", recursive=True):
if os.path.isfile(f):
try:
s = open(f, "r", encoding="utf-8", errors="ignore").read()
vals += [int(x) for x in re.findall(r'"reasoning_output_tokens"\s*:\s*(\d+)', s)]
except Exception:
pass
plt.hist(vals, bins=200, range=(0, 5000), weights=[100 / len(vals)] * len(vals))
plt.xlabel("reasoning_output_tokens")
plt.ylabel("%")
plt.show()Now I'm kinda thinking of trying per token for both, using GLM 5.2 on Fireworks for most tasks, shelling out to the big boys only when needed. Not totally confident I'll break even though.
This is the biggest downside of GPT; thinking is encrypted, so it's more of a black box than kimi/glm/deepseek. You still get thinking summaries though. It's awkward, but workable.
I do appreciate that codex is open source generally, but I don’t think it matters for this class of issue as the model is closed still
codex exec --json --skip-git-repo-check --ephemeral -s read-only --disable memories -m gpt-5.5 -c model_reasoning_effort=high "Do not use external tools. A black bag contains candies with counts: round apple 7, round peach 9, round watermelon 8; star apple 7, star peach 6, star watermelon 4. Shape is distinguishable by touch before drawing; flavor is not. What is the minimum number of candies to draw to guarantee having apple and peach candies of different shapes, i.e. round apple + star peach or round peach + star apple? Give reasoning and final number. The local project dir is irrelevant for this task, do not consult it. "
2. 516, 27
3. 516, 12
4. 516, 21
5. 516, 21
This means that the whole time we've been paying for a product that was silently routing to something completely different and inferior from gpt 5.5
Also I read through the github issues and it seems like they closed a previous issue without addressing it ???!!
whooo boy somebody from OpenAI is getting fired over this if not a class action lawsuit is almost guaranteed at this point.
I wonder if it's somehow getting confused between what's supposed to be an intermediate update vs the final result.
[1] https://github.com/openai/codex/issues/30364#issuecomment-48...
I'm also afraid to lose my "spot" if I leave codex and 5.6 is coming out so...
> OpenAI engineers earlier this month told some colleagues they had figured out a way to more than halve the cost of inference, or running existing models, thanks to some newly-discovered optimizations, according to a person with knowledge of those discussions.
I don't find "usual user psychosis" particularly fair or tasteful anyhow. You're not left with much more than subjective judgement and speculation/suspicion when all you have is a magic sink of an API endpoint that ingests your context window then spits back a continuation of it. Even if you have a standardized model test suite, claiming a stealth nerf remains an exercise in mind reading (of the people working there). Model quality can degrade without an explicit intention that way, or a downgrade of the underlying infrastructure, after all.
Being tongue-in-cheek conspiratorial, or even actually entertaining the possibility of a nerf, is no psychosis anyways. Not a fan of this trend of people abusing psychology diagnosis terminology like this. I'm sure there are people who go a step beyond and are overconfident in these judgements, maybe in their case it holds. But then that's a minority, and so what you have then is a hyperboly. Doesn't serve anyone.
Users are completely incapable of objectively evaluating model quality over time.
Which makes it all the harder to notice actual "stealth nerfs", misconfigurations or other technical issues. Because "they made the model DUMBER, for REAL this time" is background noise.
Cheap and efficient enough I could afford to use it on basically everything unlike 5.5 or Opus, but still pretty good, I preferred it to sonnet
reasoning_output_tokens count percent
━━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━ ━━━━━━━━━
0 873 28.5948
───────────────────────── ─────── ─────────
8 64 2.0963
───────────────────────── ─────── ─────────
9 60 1.9653
───────────────────────── ─────── ─────────
11 54 1.7688
───────────────────────── ─────── ─────────
516 48 1.5722
───────────────────────── ─────── ─────────
12 45 1.4740
───────────────────────── ─────── ─────────
10 43 1.4085
───────────────────────── ─────── ─────────
17 40 1.3102
───────────────────────── ─────── ─────────
13 38 1.2447
───────────────────────── ─────── ─────────
14 36 1.1792
Some were definitely below par, and I recall having to iterate on the generated code more than I wanted to. However, it is only true for a very small number of conversations.
So we're looking at a small set of affected conversations, and even within that small set, only a few will have degraded output, likely because the model can compensate for the reasoning defect over the long conversation.
There seems to be something wrong with the "commentary" channel related intermediate updates, maybe the model gets confused about what's an intermediate update vs what's the final answer? [1]
[1] https://github.com/openai/codex/issues/30364#issuecomment-48...
It could be a very dishonest way of scaling to demand during peak hours. I know that some people already scoff in this topic about the subjective nature of perceived performance of models. But the model seemed less smart when US comes online (at least from my testing over the month of May).
On my company blog post from a few weeks ago I felt the need to point this out because it had a perceptively more consistent pattern during those overlap times. Should have saved the session logs for further analysis https://webesque.agency/blog/2026-06-19-llms.html
If they managed to put together some dirty hack that lets them generate about 512 tokens worth of reasoning in parallel instead of in sequence? That would explain it.
This was past month:
516 + 518*n
516 n=0 count=4454
1034 n=1 count=318
1552 n=2 count=129
2070 n=3 count=56
2588 n=4 count=35
3106 n=5 count=14
3624 n=6 count=6
4142 n=7 count=4
4660 n=8 count=6Interesting. So 516 probably means initial 512 byte buffer and a 4 byte header. Then 516 + 518 = 1034...so another 512 + 4 byte header + 2 bytes for a linked list ref or similar, 1034 + 518 = 1552, etc.
https://github.com/openai/codex/issues/29353
What even is the point of a public-facing bug tracker “for devs, by devs” when this is how reports get treated? Might as well use Apple’s Feedback Reporter that routes to /dev/null instead.
Anyway, I find it near impossible to see how this wasn’t already caught and flagged internally – it’s not a subtle pattern. Certainly they are at the very least collecting and graphing reasoning tokens vs model vs effort” and such an obvious spike at (multiple) single stops (not even distributed over a narrow range) should have been an immediate statistical red flag… which leads me to believe (combined with the fact the previously reported issue was closed without comment) that they’re at least internally aware of this behavior even if it’s not necessarily an intentional side effect of some internal forcing metric.
There is a probably related difference of CTX windows being different (and significantly larger on PLUS) between the plans, as well as an ominous and not-user-controllable flag `use_responses_lite` which is set to true on PLUS. These two discrepancies between the plans may or may not be the cause for the issue.
Amending the base prompt shows some improvements using the canary test prompts, NOT so during real world tasks however.
The API presents mixed results: from zero failures to 2/5 failures across runs.
These results have been taken from deterministic tests and analysis of historic data on both PLUS, PRO and API accounts. More details visible on the related GH issue in comments.
I think it is very interesting that: A. Removing the section in question seems to greatly fix performance on the benchmark candy question. B. Removing the section does not appear to change at all mean reasoning token use or the 512 reasoning token hit problem
Honestly, I think this is a really cool sentence. Imagine going back to 2021 and telling someone this was a legitimate complaint about a pretty cheap and very prevalent technology in 2026.
The thing that I ask it to do is like take X and Y paper into Z paragraph --- a not-so-silly model should think of how information in X and Y are related and how they support the whole article to synthesize this sentence in a way that is coherent to the article, but 5.5 now will just copy the stuff without any reasoning about the relation. Of course, this will cost a lot of tokens and will be obvious if not done. One clear indicator is that in a few rounds you can see the length of the article get bloated to 2-3x undesirably long, which is clearly because it is not analyzing/synthesizing the info.
But on the other hand, I've been using 5.5-high on a daily basis in multithreading workflows, i.e. in parallel. I'm barely exhausting my weekly limits. I can't even Human-as-a-Service fast enough to catch up and read all the plans and implementations it does. So there is that.
I am 100% sure that I get far more value from today than I did in December. The models are smarter, the limits are higher. It’s possible there’s some “five steps forward, one step back” going on, but it’s hard to imagine complaining about that step back.
People talk about it all the time. Just check some of the dozens of forums where its non-stop complaining about nerfs, limit nerfs, performance issues etc...
Is hard to prove that any downgrade is a effect of being deliberately served a lower class model / lower quant, or whatever. Or the "optimizations" hurting the models performance.
The TOS allows for those service "optimizations", so legally, nobody has a foot to stand upon. Like when OpenAI or was it Anthropic played with the cache, this to free up more server resources, only to later discover that its gutted the long term context behavior, and heavily degraded the models as context grew.
If you want 100% guaranteed the same performance/behavior, you need to run a model yourself (be it rented GPUs online or your own local setup). But its going to cost you a lot more ...
Codex is a harness, while GPT-5.5 is a model. The last codex-branded model was 5.3. Codex as a harness ships as a CLI, a desktop app, and a web product (and I'm not at all sure how similar the underlying harness is between them.)
Is the bug here supposed to be with the CLI harness, or the model? Does it also happen in pi, opencode, etc while running GPT-5.5?
anthropic at least does have a latest and stable channel, as the other day they pushed something irritating that would skip question asking phase if you didn't reply in 60 seconds, and it broke my multi terminal workflow. like I don't know what their product people are thinking when they push this kind of stuff, but it made me switch to stable
GPT-5.5 Codex model exhibits a clustering phenomenon in which reasoning_output_tokens cluster at fixed values spaced 518 apart.
These stuck responses at fixed thresholds are strongly correlated with errors in complex tasks.
Observed phenomenon is specific to GPT-5.5; it is much less prevalent in GPT-5.4 and almost absent in GPT-5.2 and 5.3
I remember GPT 5.2 Codex being fine...
More now than ever (since original ChatGPT release), the OSS models and open harnesses (eg Pi) are looking mighty attractive.
But I’m reminded of ~2008 and the rise of “the cloud” as a marketing term that seemed to me to be a cover for dropping an expectation of rich clients, increasing a companies margins around subscriptions that would chip away at local ownership.
Then I got offput by the zealotry and absolutism around “true FoSS”, told myself I was young and moved on.
And really, a lot of subscription models I kind of can appreciate/ tolerate. Might be irksome but whatever, I get that software is expensive to make and it’s not fair in 2026 to value a yearly upgrade of Photoshop at $200. The capricious UI changes to things that’ve worked for 20 years and they take away say the classic color swatches altogether - silly and dumb.
I can use another professionally necessary tool I pay $200/ mo for, Codex, to whip up a classic swatch plugin.
Is that $200 a fair price for my token usage? I think an extremely heavy month I might’ve used a billion tokens?
But that right there is the problem. They have no idea what, specifically, profitability looks like and are going to be pulling endless levers for … I genuinely have no idea how long - at least through 2030/2032 if we tea leaves their debt obligations?
I don’t want to think about any of that. At all. I don’t want to spend time evaluating model preference and degradation and updating the nuances of how I “speak” to an AI because there’s some mystery backend experiment running on the output I use to produce functional outputs — ie the actual products I get paid to build/ maintain.
AI’s something between a tool and coworking companion, and the capricious “personality” changes due to playing with poorly understood and knobs and levers at the inference level - is maddening. To that end, I want a box in the corner I can point to and know exactly the quality of outputs that no one but myself modifies.
There has been a step change... in the amount of whining and complaining coders exhibit lately.
Firstly, we know Anthropic has been doing prompt injection into their 1P APIs (not bedrock/vertex AFAIK) for at least a year now. https://old.reddit.com/r/ClaudeAI/comments/1f6hcwo/injection...
This can be verified pretty quickly like OP — count the token metrics, if your context contains classifier-firing terms, you’ll see input_tokens being higher than your input.
So if they’re already doing that, what makes you think it’s just a dumb API, instead of a complicated pipeline filled with trade secrets and optimisations?
I just started using OpenRouter for some control testing of local models and what surprises me the most isn't that there are different providers providing different quantization levels, that makes sense, but I can't seemingly find a way of seeing what provider+model+quantization is actually used?! https://openrouter.ai/models shows the models, then say https://openrouter.ai/moonshotai/kimi-k2.7-code shows the providers but when I go to https://openrouter.ai/moonshotai/kimi-k2.7-code?endpoint=e7a... for example, why on earth is it not showing the actual details about the actual weights they're serving?! Give me details! It does have a "Precision" value that is sometimes filled out, but that seems to be a guess at best, even providers with the same values there have wildly different quality responses.
I like the idea about OpenRouter but holy hell does the implementation seem very far off from what it needs to be, in order to be useful.
There might be something in the response, or in a followup API call for the session, that you get better details. I think I've seen the details in the dashboard, so they do exist.
> With OpenAI, at least my gpt-5.5 is the same as your gpt-5.5.
How do we know that? The "orchestration" layer probably forwards to different levels of quantization. And it seems tempting to make some sort of load balancer with adaptive computation effort.
Quants in popular local inference apps (Ollama, LM Studio, etc) are the worst possible quants (RTN).
When I agree with the data: I will boast about the victories of science and empiricism, we found the perfect set of natural abstractions that are necessary and sufficient to map out the territory that carve at the joints of the problem, any concern about assumptions is rebutted with generic "Well, we're just pragmatists; we're not perfect, but clearly we're converging on the right direction! You're clearly someone who just wants to nitpick and not get any work done."
My experience with certain hackernews commenters in a nutshell.
"There's no evidence to prove xyz" then they would say, as your evidence was never as rigorous as theirs. And since they were proclaimed to be the only authorized scientists in the room, by authority of big governing bodies, they were right.
So people will see whatever evidence they want, and whine and complain to dig into their side as tribalistic creatures.
I'm curious if you wrote this or had a LLM write it.
I'm genuinely curious to be clear as I don't see why anyone would bother to go through a LLM to write such a short reply. Have we reached the point where Claudeisms that are this obnoxious have become part of regular speech?
[0]: https://en.wikipedia.org/wiki/Joke
(…just like that)
"A little secret. About 5% of our production traffic is on the Pi harness, about another 5% is on OpenCode. Reminder you can use your ChatGPT account in a flourishing set of other tools.
We’ll continue to make Codex awesome, but you have options."
Claude web definitely feels like it's the same models behind as in API, with much less extra behavior/layers that make it behave differently.
I think they even partnered with opencode or something like that (don't remember).
OpenAI officially allows that with subscriptions.
OpenAI's harness is fully open source[1], and (AFAIK) doesn't come with any kind of signed-build request integrity verification like Claude does. And by that logic, if you're allowed to use their API with a fork of Codex that you yourself compiled, there's nothing stopping you from making some other harness act like such a fork.
Warning: I use this daily myself for work, but beyond me arguing with it to get the CDP architecture, this was completely vibe-coded. I mostly use vim mode (jump to line, jump to prev/next user message) and the account switcher.
Didn't the foundries take action against those in the past? I don't see "delve" nearly as often anymore. Why are the models spiraling like this now?
What really grinds my gears is the constant need to guess what I'm doing and offer a million random follow-ups. I asked what the weather was like, I don't appreciate the 5 paragraphs of tokens burned on weather-appropriate activity suggestions.