> Knowledge and reasoning are inextricably interwoven in the weights of the neural network—there is no independent, explicitly represented set of beliefs.

I'm not sure I understand this. Do we have evidence humans have an independent set of beliefs not shaped by knowledge and reasoning? If so, where do these come from?

I'm especially confused about a prior statement as a scientist:

> Three shortcomings prevent what chatbots do from qualifying as reasoning (in a way that a scientist might recognize).

How does a set of beliefs help with reasoning?

> Third, while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact, reaching an answer by one route but reporting another.

We also often do the same as humans.

Humans have an inherent flaw in that our brains are wired to see intelligence and reasoning where there is none. Our brains fill in data that simply isn’t there.

Folks see Jesus in burnt toast. Monet was a master of exploiting this where what’s really just blotches of color our brains fill into beautifully detailed images.

Our experience with LLMs is no different. Folks believe there is some deeper intelligence there but it’s all still just 1s and 0s on a computer chip. We’re interpreting things happening that simply are not happening.

> while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact, reaching an answer by one route but reporting another.

Doesn’t research show humans often do this too? There’s a pretty famous paper from the 70s about that [1], and lots of subsequent evidence. We also have choice blindness [2], we confabulate reasons [3], and we even change our choices (sometimes negatively) after trying to introspect [4].

[1] https://www.researchgate.net/publication/229060046_Telling_m...

[2] https://pubmed.ncbi.nlm.nih.gov/16210542/

[3] https://pmc.ncbi.nlm.nih.gov/articles/PMC5986841/

[4] https://pubmed.ncbi.nlm.nih.gov/2016668/

My pet theory is that there are different types of intelligence that have different pros and cons. Social/cultural, intuition, and structural.

Structural is like step by step reasoning or math or raw compute.

Intuition is statistical from repeated trial and error.

And social is leaning on the wisdom of the crowds. So like high latitude countries where they eat fish for breakfast and get better health outcomes.

So I believe that LLMs have stumbled upon a partial component of our social intelligence. Word distribution, ontologies, jargon, information theory (frequently used symbols should be short). We mutate the language that we speak to be useful to us based on the problems we face. To some extent being able to talk the talk means you can also walk the walk. At least partially.

It's kind of shocking how far they can get, but at the same time it's kind of a surprise how far they don't. The existence of agentic harnesses is sort of an admission of defeat.

While some might be fooled into thinking that they reason, everyone I've met isn't. As a software engineer I'm drowning in work. And if that's not an admission that this isn't a real intelligence then I don't know what is.

But ultimately it looks like we've got all the individual components sorted. The old school 70s era stuff has a lot of the structural intelligence covered. The data science era of statistical ML has the intuition. And LLMs have the intelligence from our culture.

Maybe there are more general or energy efficient or powerful or special purpose techniques out there. And maybe combining everything together requires some additional insight. Regardless it feels like moving forward to something better than our current AI landscape is plausible, albeit with a completely unknown level of effort.

Dont be fooled. Reasoning does not happen in the prediction of the next token, it happens virtually in the text that is created.

The next token prediction is just "the hardware" following the underlying rules. Like the basic set of rules.. in a sense similar to how the "game of life" does not really contain gliders. Gliders are just a self stabilised system that arrises from the simple rules.

I don't know what you would call it, but reasoning/thinking/whatever it is, is a way for LLMs to tighten their sampling space, while allowing for wide sampling to still happen. This is akin to people brainstorming ideas.

I know that sounds confusing, let me break down how I think about this.

1. LLMs don't pick the token that ends up being used. This is by design, if the LLM gives a wide choice, it can better adapt to real world scenarios. i.e. generalize.

2. Without reasoning, this means that the LLM either locks in on whatever the sampler picked. Or decides mid-sentence/response to correct itself. This is what used to happen before reasoning, still happens if you turn reasoning off.

3. With reasoning, the LLM can make as many mistakes as it wants and explore its sampling space. Then use its vast pattern matching capabilities to decide which parts of the reasoning make sense and which were idiot ideas.

4. Enabling reasoning makes it so LLMs are much more confident on the final response, and the logits should theoretically all be near 99% on a single token for every token, i.e. much closer to greedy decoding. It analyzed all the possible options and figured out the best outcome, so a stray sample doesn't cause the answer to go awry.

This is why reasoning traces are filled with "but wait". I don't know if those were added in organically or artificially in the RL training, but regardless they're a good way to let the LLM keep generating other options and explore it's sampling space to the fullest.

Note: I haven't tested any of this and it's just my theory, but I'm sure if you really wanna know you can have claude run some smoke tests :)

Well, it's not doing hard reasoning like Lean or Prolog would do, but it does an approximation of that, as it was trained to reproduce linguistic patterns that encode reasoning.

Jet planes don't fly by flapping their wings...

People should read the article instead of responding with what they believe to be clever quips. The article's author gives a very good argument for why what LLMs are doing in their chain of thought is not reasoning.

This is an ad for the new startup by the author, who will now focus on reasoning models. His insight seems to be that LLMs should make their assumptions explicit first, then map out a search space and provide reasons for why they should take one path or the other.

I think it's an interesting approach but the overall discussion about reasoning is really pedantic. What the author is describing here is one approach out of many, and in my opinion it doesn't cover what humans colloquially think of when they hear reasoning (while the output from a chain of thought sometimes does).

Law of headlines. Past the slightly inflammatory framing, the article actually makes a case for making LLM reasoning more rigorous. Which, you know what, is absolutely something you could train them on. Might be worth exploring if we can apply the rigor rigorously. You could have smaller models that converge at all on harder problems, and larger ones that converge faster.

well, its a matter of definitions.

the categories i like to use to describe what llms are capable and incapable of are: instrumental reason, which is reason as a tool for achieving a goal; and objective reason, which is reasoning about which goals are good or bad, or worth pursuing.

llms are, i think, approaching or have achieved better-than-human performance on the former category in a wide variety of applications.

the latter, not so. leaving aside that there are schools which claim (dogmatically, imho) humans don't or can't engage in objective reason, i dont believe llms are structurally capable of it. their goals can only be imposed on them from outside, coming from prompts, implicit value assumptions in training data, loss function, and rlhf. there is something about human interior experience of an objectively existing world that lets is evaluate true/false/good/bad in a way that is unique to humans among other animals.

llms can't do it. i dont just mean on ethical, epistemic, or aesthetic judgements, but even in practical circumstances like the ones engineers encounter. the reason engineers still have to work alongside llms, even though lllms are (imho) far better programmers and technicians, is that even when given a goal, there is always a graph of evaluations that lead to that objective and llms routinely fail to evaluate the tradeoffs and land in states in outcome space that are subtly (or not so subtly) wrong, even though the objective is complete!

forgive typos, i am on mobile.

An LLM is just a brain in a vat. Our brains would also not reason in such a condition. Given the right framework that can do miraculous things. I am not saying that these systems are conscious, but they are able to abstract their context in a way allowing them to understand their own limitations.

https://www.finextra.com/blogposting/31255/adaptability-as-e...

I thought I saw a paper recently explaining that LLMs have a global workspace. Is that not like having the internal state that he's talking about?

I disagree. They do reason during the reenforcement learning stage. They don't reason at inference. A good metaphor is that useful output are like nuggets that exist after reenforcement learning which need to mined to be, in LLM talk, "surfaced." Without supervised fine tuning, the reasoning models will add weight to tokens, words and phrases like "verify" and "check work" which will cause it to follow those verifying tokens with reasoning tokens that do just that, verify.

<unbearable web page to read>

Don't be fooled, submarines don't swim.

I wonder if as a hack, some of the shortcomings mentioned could be addressed through prompting.

E.g. "Approach this problem iteratively. As you form a hypothesis, track the confidence you have in various explanations you're considering, what evidence you're weighing to support each, and the unresolved questions you're holding onto. Log all that for later inspection.

Be methodical when evaluating evidence and only accept facts you have verified. At every stage, gauge how much each possible next step resolves uncertainty, and discard options unlikely to advance progress. Divide the functions I described into subagents responsible for each, and coordinate with them as you work."

If AI cannot reason, can humans?

I think the real question is: how do we define reasoning?

My view is that AI can reason, just differently from humans.

> The gains have proved real, above all in mathematics and coding. But unlike AlphaGo’s search, this does not introduce a genuinely separate reasoning mechanism: The intermediate reasoning is still produced by the same next-token prediction process, iterated for longer before the model commits to an answer.

But aren't external tools used, which implement hard reasoning? Like in the case of mathematics proof work, external theorem provers?

The LLM is literally not doing "thinking"; it's just throwing shit at a theorem proving wall, until some of it sticks.

This is the double edged sword of calling it AI, of using terms like Temperature and Hallucinate and Thought.

Stop trying to compare either system to a human and look at it for what it is -

A prediction engine that runs fast enough to brute force problems.

In the case of alpha go its "innovation" was millions of games played against itself. It had bound parameters and strict win conditions.

In the case of LLM's you can deploy 1000's of agents to smash themselves against an idea. The whole hugging face attack is an example of this (1200 agents out of an unknown number chose that path).

There is the old saying about monkeys, typewriters and Shakespeare. Well we have better monkeys who basically follow a derivative of zipfs law (not actually), who use tokens not letters and their goal in many cases is testable (compile, unit, E2E).

But neither do the vast majority of humans.