Every field has terms of art, and 'reasoning' is one for LLMs. But that doesn't mean it has the same properties as 'reasoning' in other contexts, because you're not referring to the same thing.
Why doesn't my asteroid belt buckle?
> Our findings consistently challenge the prevailing narrative that intermediate tokens constitute a semantically meaningful reasoning process. First, we observe a pronounced lack of correlation between solution correctness and trace validity—models frequently produce invalid reasoning traces even when they arrive at correct solutions. Second, and more strikingly, models trained on corrupted or semantically irrelevant traces achieve performance comparable to, and often exceeding, that of models trained on correct traces, especially on out-of-distribution tasks.
Related: Poster side dialogue and Q&A about this work at ICML. Very good. https://news.ycombinator.com/item?id=49277303
Because the field of artificial "intelligence" is constructed around the idea of applying psychological metaphors to computational systems (a very powerful idea!) it's almost a worst case scenario for these people.
Suddenly, they're reversing the metaphors and applying computational schema to psychological processes ("aren't we really just stochastic parrots ourselves?!"); or, like here, they find themselves surprised and confused when they stumble across the natural boundaries of the metaphor experimentally.
It's because they never had sight of the boundaries in the first place and maybe never can quite see them. The words only make sense to them as literal equivalence, and so their surprise when they run into stuff like this is earnest and deep.
I think you certainly could look at LLMs as "thinking" metaphorically, but I also don't think it is necessarily only a metaphor.
> models often determine their answers based on implicit biases tied to question templates, then construct reasoning chains to justify their predetermined conclusions > its reasoning was correct right until the final step (Yes/No answer)
When doing the paper we noticed that models are very good at generating post-hoc plausible CoT, which to me knowledge can happen quite often with relatively easy tasks.
You might be interested in reading this other paper that came out after ours: https://arxiv.org/abs/2507.05246
It is annoying when the bot seems be “reasoning” correctly and then makes an obvious mistake at the end. And perplexing when it seems to be completely wrong and then pull the right answer out of a magic hat at the end.
I guess it makes sense; the “reasoning” steps aren’t actually doing logic, just adding more context to influence the final generation, right? But it is weird to see.
From March last year: https://transformer-circuits.pub/2025/attribution-graphs/bio...
There's no reason to believe the model's self-reported "thinking" bears any relation to the mechanics by which it arrived at some output.
if not, then there's certainly some bearing, but not necessarily in how we read the tokens as text
This points to a hypothesis that the content of the tokens is only slightly related to the mechanism by which it improves performance, and that primarily the extra tokens allow the original prompt to be processed more deeply by the model, because earlier tokens will essentially pass through the model many more times than later ones.
While maybe that boundary can be fuzzy, we're unequivocally and deeply in "metaphor" territory here.
It's a relevant comment in this instance because we're discussing concepts you need to be both trained and practiced in to reason about, and that our discipline has traditionally been blind to. Plenty of people working with LLM context issues who've never been exposed to the idea of 'subtext' or could tell you why it would matter to their direction of effort.
how much of the training data had thinking traces that dont make sense to people as being actually a description of why the output should be that way?
A naked "natural intelligences do this too" might be a bit too short to be useful. But if we can add when/where, cite papers, or show ways in which the parallel operates, then it might be useful.
Compare, eg, talking about a robot arm, and someone goes "a natural arm does this too". You can tell about the fact that it has the same degrees of freedom in the same places, or how this pertains to inverse kinematics, or etc...
Same way here, "this happens to be how natural intelligences seem to solve this too! According to Foo, Bar, Baz et al (2026) the gadget is always twiddled beforehand in macaque apes. " or "Same for natural intelligence: I've noticed I use the same general algorithm myself. I've always considered this the correct way to do translation between languages". --
A more concrete example of a useful answer here.
Natural intelligence does this too! When given the question "explain your reasoning" humans are indeed quite prone to post-hoc confabulation. [1]
[1] https://home.csulb.edu/~cwallis/382/readings/482/nisbett%20s... "Telling More Than We Can Know: Verbal Reports on Mental Processes" (this citation is quite old and may have been superseded, mostly just to illustrate how the rule might work)
In other words, a lot of this depends on what scale/scope is being inspected. On the high level, the world is chaos rather than a meticulous and inscrutable plan of the Illuinati Shadow Cabal. On the low level, people do things for reasons, even if they're dumb ones.
With respect to the "current political climate", I'd like to suggest that a lot of dumb or seemingly "against their own interests" stuff is due to people prioritizing costly in-group loyalty signals. Their interest in staying good with the tribe is just higher than their interest against a dumb national policy.
Project 2025, Maralago accords.
So far the only major policy item the Trump admin seems to have not intended was the Iran War. Israel killing the intended replacement, Iran leveraging the straight of Hormuz, and dropping three Tomahawks on an elementary school really botched that one.
It's just peter theil's texting groups.
The ability to conspiracy both willing and unwilling is such a low threshold now, it's virtually indistinguishable.
You watch one billionaire do something and you're like, I'm a billionaire, I should do that too.
The fact that there's so few billionaires, the probability that they conspire together both direct and indirect approaches 1.
The inverse of course is rediciously hard to conceive: the working class bands together to get something like universal healthcare.