It's safe to say that argument has collapsed utterly. Modern LLMs virtually never collapse into endless babbling loops. Far more importantly, they solve objectively hard problems. An LLM didn't merely inspire a mathematician to spot a counterexample to the Jacobian Conjecture that had been in front of his nose the whole time; it found the counterexample. These things are unquestionably generating meaningful outputs.

(That obviously doesn't mean that you can simply trust LLM outputs!)

At this point, trying to dunk on frontier models by calling them "stochastic parrots" is saying more about you than about the models. Moreover, it suggests you're not only out on a limb about the capabilities of LLMs, but also on what even the skeptic literature about it was saying.

You should know what "stochastic parrot" actually meant; you shouldn't be using the term just because you think it sounds snazzy.

All we have to do is assume that things progress in a linear way.

Do you remember GPT-2? It was a bizarre curiosity, but was nevertheless considered a huge advance in late 2019. Practically useless, but interesting.

It grew, and three and a half years later we got GPT-4, which was exceptionally capable on release.

It grew still further, and three and a half years hence we now have GPT-6. This is already more intelligent, more imaginative, more disciplined, and far more erudite than the vast majority of individual humans. If you have an intellectual skill -- if you are, say, a mathematician or a chemist -- GPT-6 also has that skill and is functionally your peer. And then it has many other skills that you lack; if nothing else, it is superhuman in breadth.

Assume a steady rate of progress. In three and a half years, we're going to have something that is genuinely a general superintelligence, and we're probably going to see recursive self-improvement begin to kick in (though subject to physical constraints on compute, etc.)

Now is the time for an RSI start-up, I guess!

Anyway, all you have to do is assume that things don't plateau. They have not plateaued to date; if anything, progress appears to be accelerating, and people are starting to panic that there are no brakes on this thing.

However, what seems to still be lacking in LLMs — aside from plasticity (continuous learning), perception of time, and anything related to emotions and desires (which underly motivations) — is various types of awareness, and sound judgement. We are still far from a good understanding of these aspects.

It's fascinating the hills people will die on. Let's even set aside the fact that LLM internals are mostly opaque but logical branching points ? Look at the Watson test. Most people fail a simple conditional reasoning problem unless it’s dressed up in familiar social context, like catching cheaters.

Where does this idea of general intelligence as logic automatons actually exist ? Because it's sure as hell not real life. Humans are not like this, Apes are not like this, Birds are not like this, Cetaceans are not like this. Fiction. Fiction is the only realm this reality of GI exists, so it's patently absurd when people hide behind it like anything that doesn't present as doing it must be obviously wrong.

I often hear this asserted, but never with any actual justification.

Why can it not?

Everyone who rejected the stochastic parrot argument has been right about AI progress and the increasing abilities of LLM's in math, coding, writing, etc. You would be laughed at in 2020 for claiming an LLM could write coherently or could possibly pass the Turing test.

Those who claim that AI aren't capable of "genuine judgement" or whatever rhetorical flourish they feel is most effective at dismissing AI capabilities would tell you in 2022 that AI would never be capable of coding, or in 2023 it would never be capable of mathematics beyond grade-school level. Those same people now will claim that AI could never come up with a genuine breakthrough in math, science, etc. Detractors of AI have a full time job of moving goalposts.

That being said, AI, once sufficiently powerful, will grant whoever controls it the ability to mold the world as they see fit, for better or worse. They will have access to superhuman mathematicians, coders, strategists, PR, financial engineering, legal skill, etc. Essentially an army of superhuman intellects which can afford them guaranteed success in any domain they wish to pursue. Anyone who doesn't see this as potentially the most destabilizing event in human history likely isn't thinking deeply enough about it, or is still knee deep in denial.

(1) A far-reaching tendency to overextrapolate from the low-hanging fruit of the last few years of pretraining progress. GPT-2 to GPT-3 may have been a quantum leap, but GPT-3 to GPT-4 was not, and GPT-4 to 5 even less so.

The party has been kept going by RL and agents, but still, there is indeed a point of diminishing returns, not just relative to available compute but to how much training is possible when the entire intellectual output of humanity, plus a raft of synthetic data, has already been inhaled by the training process.

If one is to internalise the things that are said here on HN with regularity about model progress, and sentences ending with "yet" or "for now", then it would be easy to conclude that my 10 year-old son, who gained 3 inches of height last year, will be tallest structure on the planet by age 17.

(2) Inability to distinguish between technological, computational, and energetic limits of LLM capabilities vs. ontological / conceptual ones. There are some things LLMs cannot do, or at least do well, at any size, at infinite size and with infinite compute, simply due to the very nature of what LLMs are to begin with.

This latter topic receives almost no attention, except maybe from Gary Marcus and Yann LeCun. In that respect, this article is a breath of fresh air, insofar as it highlights that LLMs aren't "AI" at all, as we have traditionally understood the concept.

They really _are_ stochastic parrots. The relevant questions are about how much that matters for some domain or set of applications, not whether they are an emerging alien intelligence with civilisation-threatening capabilities.

That proves the fact that the audience is not really agentic (not the ones that can cause or influence things). The agentic nature in people was killed long back with all education, rules and pervasive reach of the state into the lives of people. Common people are just subjects without an intention and will at this point. They are like those leaves that float in a stream, while the stream finds its way.

At best, all that a today's average person can do is, post sone interesting post or comment on HN, or the other social media and count their likes, like I do now. There ends their agentic effort.

How can you possibly hold this belief after Tuesday? An LLM solved a significant number of major open problems in many subfields of mathematics, including a major breakthrough on the Riemann Hypothesis. Get your head out of the sand.

This statement belongs to the same epistemic class as Moon-landing denial. Flagged for kookery.

<3

This sort of nihilism/defeatism always irks me. It's so selfish, so privileged.

Tell my starved, murdered ancestors "let us enjoy the end" of their era of stability's denoument.

And, truly, it is not the end. Inconveniently, many—if not most—of us will survive. Will continue on. And it's for the betterment of those who come after us that we ought to toil a little, and not indulge in this childish, self-interested to nihilism.

marginalized populations have repeatedly survived past "apocalypses" not through aesthetic surrender, but through the deliberate, daily reproduction of the commons, ie through pooling resources, sustaining life for its own sake, and preserving collective memory.

Yielding to fashionable doomism merely validates the oppressor's narrative that the future is already decided. A superstition of doom is no different from a superstition of inevitable progress; both models rely on an abstract determinism that strips humans of agency, that encourages the us to accept suffering as unalterable destiny or "the will of God" rather than an artificial, historical barrier.

So, I agree with a lot of what OP is saying, but I think they're a jerk.

Not anything interesting here that hasn't already been stated 20 other times.

He is right that the AI giants would like a bit of regulatory capture.

How can we be so confident that there is anything LLMs fundamentally cannot do? Proving that impossibility seems hard.

Drawing trend lines far into the future is foolish, but the recent trend is clear and its not obvious how much more progress is needed before they start having a meaningful impact on more aspects of life.

> not whether they are an emerging alien intelligence with civilisation-threatening capabilities.

The people building them are explicitly attempting to do this. They may not succeed, but what if they do? Seems worth considering that scenario.

> The dangers posed by this technology on its present trajectory are best understood as simple accelerants for the existing dangers humans pose to themselves.

Is very true. All the doomspeak "aI EsCaPeD AnD HaCkEd aNoThEr sYsTeM" was a human error all along, and the tech was doing what it is programmed to do, retry until a satisifiable outcome is met.