What distinguishes human intelligence from LLM intelligence is largely information that the latter hasn't mopped up. So the information that the former is uniquely privy to is what gives it economic value.
In order to predict the next token in this line "The solution to your problem is..." the entity needs to be able to solve your problem. Which is well beyond what people think of when they think of a token predicting statistical sequence.
The statistical sequence description is ridiculously incomplete.
People have fixed software bugs with zero knowledge of what the code is doing and no knowledge of the programming language at hand, purely by statistical pattern matching. Solving the problem with no understanding of it beyond the idea that whatever it was, the right answer was probably something that kind of matched the kind of patterns found in things they'd seen before.
So what does it mean, "to be able to solve your problem"?
Is that even true? Beyond an extremely simple example?
It is technically true but meaningless.
Anytime you hear someone saying "it can't do this, or that..." they are factually incorrect.
The denial around these issues is beyond creepy: AI has clearly hit some tender nerve in the human mind.