Care to expand on how his theories can be taught in such a binary way?

Same thing happened in Astronomy. Students of Fred Hoyle can't work in some institutions. &c &c.

[1] https://smbc-wiki.com/index.php/How-math-works

Chomsky talks about how the current approach can't tell you about what humans are doing, only approximate it; the example he has given in the past is taking thousands of hours of footage of falling leaves and then training a model to make new leaf falling footage versus producing a model of gravity, gas mechanics for the air currents, and air resistance model of leaves. The later representation is distilled down into something that tells you about what is happening at the end of some scientific inquiry, and the former is a opaque simulation for engineering purposes if all you wanted was more leaf falling footage.

So I interpret Chomsky as meaning "Look, these things can be great for an engineering purpose but I am unsatisfied in them for scientific research because they do not explain language to me" and mostly pushing back against people implying that the field he dedicated much of his life to is obsolete because it isn't being used for engineering new systems anymore, which was never his goal.

It's completely irrelevant because the point he's making is that LLMs operate differently from human languages as evidenced by the fact that they can learn language structures that humans cannot learn. Put another way, I'm sure you can point out an infinitude of similarities between human language faculty and LLMs but it's the critical differences that make LLMs not useful models of human language ability.

> When you feed them “impossible” languages (e.g., mirror-order or random-agreement versions of English), perplexity explodes and structure heads disappear—evidence that the models do encode natural-language constraints.

This is confused. You can pre-train an LLM on English or an impossible language and they do equally well. On the other hand humans can't do that, ergo LLMs aren't useful models of human language because they lack this critical distinctive feature.

It certainly teaches us many things. But an LLM trained on as many words (or generally speaking an AI trained on sounds) in similar quantities of a toddler learning to understand, parse and apply language, would not perform well with current architectures. They need orders of magnitude more training material to get even close. Basically, current AI learns slowly, but of course it’s much faster in wall clock time because it’s all computer.

What I mean is: what makes an ALU (CPU) better than a human at arithmetic? It’s just faster and makes fewer errors. Similarly, what makes Google or Wikipedia better than an educated person? It’s just storing and helping you access stored information, it’s not magic (anymore). You can manually do everything mechanically, if you’re willing to waste the time to prove a point.

An LLM does many things better than humans, but we forget they’ve been trained on all written history and have hundreds of billions of parameters. If you compare what an LLM can do with the same amount of training to a human, the human is much better even at picking up patterns – current AIs strongest skill. The magic comes from the unseen vast amounts of training data. This is obvious when using them – stray just slightly outside of the training zone to unfamiliar domains and ”ability” drops rapidly. The hard part is figuring out these fuzzy boundaries. How far does interpolating training data get you? What are the highest level patterns are encoded in the training data? And most importantly, to what extent do those patterns apply to novel domains?

Alternatively, you can use LLMs as a proxy for understanding the relationship between domains, instead of letting humans label them and decide the taxonomy. One such example is the relationship between detecting patterns and generating text and images – it turns out to be more or less reversible through the same architecture. More such remarkable similarities and anti-similarities are certainly on the horizon. For instance, my gut feeling says that small talk is closer to driving a car but very different from puzzle solving. We don’t really have a (good) taxonomy over human- or animal brain processes.

The same way that reddit comments arent a formal debate.

Mocking is absolutely useful. Sometimes you debate someone like graham hancock and force him to confirm that he has no evidence for his hypotheses, then when you discuss the debate, you mock him relentlessly for having no evidence for his hypotheses.

> Yet here is Chomsky addressing a lay audience that has no linguistics background

So not a formal debate or paper where I would expect anyone to hold to debate principles.

> instead of even attempting to summarize the arguments for his position..

He makes a very clear, simple argument, accessible to any layperson who can read. If you are studying insects what you are interested in is how insects do it not what other mechanisms you can come up with to "beat" insects. This isn't complicated.

  Chomsky introduced his theory of language acquisition, according to which children have an inborn quality of being biologically encoded with a universal grammar

Isn't AI optimism an ideological motivation? It's a spectrum, not a mental model.

For Chomsky specifically, the entire existence of LLM, however it's framed, is a massive middle finger to him and a strike-through on a large part of his academic career. As much as I find his UG theory and its supporters irritating, it might be felt a bit unfair to someone his age.

Usually what happens is the information bubble bursts, and gets corrected, or it just fades out.

I was quite dismissive of him on LLMs until I realized the utter hubris and stupidity of dismissing Chomsky on language.

I think it was someone asking if he was familiar with the Wittgenstein Blue and Brown books and of course because he as already an assistant professor at MIT when they came out.

I still chuckle at my own intellectual arrogance and stupidity when thinking about how I was dismissive of Chomsky on language. I barely know anything and I was being dismissive of one of unquestionable titans and historic figures of a field.

An equivalent observation might be that the only people who seem really, really excited about current AI products are grifters who want to make money selling it. Which looks a lot like Blockchain to many.

That may change, particularly if the intelligence of LLMs proves to be analogous to our own in some deep way—a point that is still very much undecided. However, if the similarities are there, so is the potential for knowledge. We have a complete mechanical understanding of LLMs and can pry apart their structure, which we cannot yet do with the brain. And some of the smartest people in the world are engaged in making LLMs smaller and more efficient; it seems possible that the push for miniaturization will rediscover some tricks also discovered by the blind watchmaker. But these things are not a given.

LLM designs to date are purely statistical models. A pile, a morass of floating point numbers and their weighted relationships, along with the software and hardware that animates them and the user input and output that makes them valuable to us. An index of the data fed into them, different from a Lucene or SQL DB index made from compsci algorithms & data structure primitives. Recognizable to Azimov's definition.

And these LLMs feature no symbolic reasoning whatsoever within their computational substrate. What they do feature is a simple recursive model: Given the input so far, what is the next token? And they are thus enabled after training on huge amounts of input material. No inherent reasoning capabilities, no primordial ability to apply logic, or even infer basic axioms of logic, reasoning, thought. And therefore unrecognizable to Chomsky's definition.

So our LLMs are a mere parlor trick. A one-trick pony. But the trick they do is oh-so vastly complicated, and very appealing to us, of practical application and real value. It harkens back to the question: What is the nature of intelligence? And how to define it?

And I say this while thinking of the marked contrast of apparent intelligence between an LLM and say a 2-year age child.

And there's a fact here that's very hard to dispute, this method works. I can give a computer instructions and it "understands" them in a way that wasn't possible before LLMs. The main debate now is over the semantics of words like "understanding" and whether or not an LLM is conscious in the same way as a human being (it isn't).

These days, Chomsky is working on Hopf algebras (originally from quantum physics) to explain language structure.

It's like wondering how well your shoes fit your feet, forgetting that shoes are made and chosen to fit your feet in the first place.

This quote brought to mind the very different technological development path of the spider species in Adrian Tchaikovsky's Children of Time. They used pheromones to 'program' a race of ants to do computation.

Chomsky made interesting points regarding the performance of AI with the performance of biological organisms in comparison to human but his conclusion is not correct. We already know that cheetah run faster human and elephant is far stronger than human. Bat can navigate in the dark with echo location and dolphin can hunt in synchronization with high precision coordination in pack to devastating effect compared to silo hunting.

Whether we like or not human is the the top unlike the claim of otherwise by Chomsky. By scientific discovery (understanding) and designing (engineering) by utilizing law of nature, human can and has surpassed all of the cognitive capabilities of these petty animals, and we're mostly responsible for their inevitable demise and extinction. Human now need to collectively and consciously reverse the extinction process of these "superior" cognitive animals in order to preserve these animals for better or worst. No other earth bound creature can do that to us.

Many of the comments herein lack that feature and seem to convey that the author might be full of him(her)self.

Also, some of the comment are a bit pejorative.

OTOH, consider LLMs as a roomful of monkeys that can communicate to each other, look at words,sentences and paragraphs on posters around the room with a human in the room that gives them a banana when they type out a new word, sentence or paragraph.

You may eventually get a roomful of monkeys that can respond to a new sentence you give them with what seems an intelligent reply. And since language is the creation of humans, it represents an abstraction of the world made by humans.

I happen to agree with his view, so i came armed to agree and read this with a view in mind which I felt was reinforced. People are overstating the AGI qualities and misapplying the tool, sometimes the same people.

In particular, the lack of theory, and scientific method means both we're, not learning much, and we've rei-ified the machine.

I was disappointed nothing said of Norbert Weiner. A man who invented cybernetics and had the courage to stand up to the military industrial complex.

But what we're good as using all of our capabilities to transform the world around us according to an internal model that is partially shared between individuals. And we have complete control over that internal model, diverging from reality and converging towards it on whims.

So we can't produce and manipulate text faster, but rarely the end game is to produce and manipulate text. Mostly it's about sharing ideas and facts (aka internal models) and the control is ultimately what matters. It can help us, just like a calculator can help us solve an equation.

EDIT

After learning to draw, I have that internal model that I switch to whenever I want to sketch something. It's like a special mode of observation, where you no longer simply see, but pickup a lot of extra details according to all the drawing rules you internalized. There's not a lot, they're just intrinsically connected with each other. The difficult part is hand-eye coordination and analyzing the divergences between what you see and the internal model.

I think that's why a lot of artists are disgusted with AI generators. There's no internal models. Trying to extract one from a generated picture is a futile exercice. Same with generated texts. Alterations from the common understanding follows no patterns.

"To characterize a structural analysis of state violence as “apologia” reveals more about prevailing ideological filters than about the critique itself. If one examines the historical record without selective outrage, the pattern is clear—and uncomfortable for all who prefer myths to mechanisms." the fake academic facade, the us diabolism, the unwillingness to see complexity and responsibility in other its all with us forever ..

> We can make a rough distinction between pure engineering and science. There is no sharp boundary, but it’s a useful first approximation. Pure engineering seeks to produce a product that may be of some use. Science seeks understanding. If the topic is human intelligence, or cognitive capacities of other organisms, science seeks understanding of these biological systems.

If you take this approach, of course it follows that we should laugh at Tom Jones.

But a more differentiated approach is to recognize that science also falls into (at least) two categories; the science that we do because it expands our capability into something that we were previously incapable of, and the one that does not. (we typically do a lot more of the former than the latter, for obvious practical reasons)

Of course it is interesting from a historical perspective to understand the seafaring exploits of Polynesians, but as soon as there was a better way of navigating (i.e. by stars or by GPS) the investigation of this matter was relegated to the second type of science, more of a historical kind of investigation. Fundamentally we investigate things in science that are interesting because we believe the understanding we can gain from it can move us forwards somehow.

Could it be interesting to understand how Hamilton was thinking when he came up with imaginary numbers? Sure. Are a lot of mathematicians today concerning themselves with studying this? No, because the frontier has been moved far beyond.*

When you take this view, it´s clear that his statement

> These considerations bring up a minor problem with the current LLM enthusiasm: its total absurdity, as in the hypothetical cases where we recognize it at once. But there are much more serious problems than absurdity.

is not warranted. Consider the following, in his own analogy:

> These considerations bring up a minor problem with the current GPS enthusiasm: its total absurdity, as in the hypothetical cases where we recognize it at ones. But there are much more serious problems than absurdity. One is that GPS systems are designed in such a way that they cannot tell us anything about navigation, planning routes or other aspects of orientation, a matter of principle, irremediable.

* I´m making a simplifying assumption here that we can´t learn anything useful for modern navigation anymore from studying Polynesians or ants; this might well be untrue, but that is also the case for learning something about language from LLMs, which according to Chomsky is apparently impossible and not even up for debate.

While there's some things in this I find myself nodding along to in this, I can't help but feel it's an a really old take that is super vague and hand-wavy. The truth is that all of the progress on machine learning is absolutely science. We understand extremely well how to make neural networks learn efficiently; it's why the data leads anywhere at all. Backpropagation and gradient descent are extraordinarily powerful. Not to mention all the "just engineering" of making chips crunch incredible amounts of numbers.

Chomsky is extremely ungenerous to the progress and also pretty flippant about what this stuff can do.

I think we should probably stop listening to Chomsky; he hasn't said anything here that he hasn't already say a thousand times for decades.

What is elegant as a model is not always what works, and working towards a clean model to explain everything from a model that works is fraught, hard work.

I don’t think anyone alive will realize true “AGI”, but it won’t matter. You don’t need it, the same way particle physics doesn’t need elegance

https://www.nytimes.com/2023/03/08/opinion/noam-chomsky-chat...

They may not be doing strict formal logic, but they are definitely compressing information into, and operating using, symbols.

LLM's seem to have proven themselves to be more than a one-trick-pony. There is actually some resemblance of reasoning and structuring etc.. No matter if directly within the LLM, or supported by computer code. E.g it can be argued that the latest LLMs like Gemini 2.5 and Claude 4 in fact do complex reasoning.

We have always taken for granted you need intelligence for that, but what if you don't? It would greatly change our view on intelligence and take away one of the main factors that we test for in e.g. animals to define their "intelligence".

Even more strangely, the act of giving a statistical model symbolic input allows it to build a context which then shapes the symbolic output in a way that depends on some level of "understanding" instructions.

We "train" this model on raw symbolic data and it extracts the inherent semantic structure without any human ever embedding in the code anything resembling letters, words, or the like. It's as if Chomsky's elusive universal language is semantic structure itself.

Chomsky vs Norvig

https://norvig.com/chomsky.html

I dunno if people knew it at that time, but those two views are completely equivalent.

Yes, because anthropomorphism is hardwired into our biology. Just two dots and an arc triggers a happy feeling in all humans. :)

> of practical application and real value

That is debatable. So far no groundbreaking useful applications have been found for LLMs. We want to believe, because they make us feel happy. But the results aren't there.

I'm surprised that he doesn't mention "universal grammar" once in that essay. Maybe it so happens that humans do have some innate "universal grammar" wired in by instinct but it's clearly not _necessary_ to be able to parse things. You don't need to set up some explicit language rules or generative structure, enough data and the model learns to produce it. I wonder if anyone has gone back and tried to see if you can extract out some explicit generative rules from the learned representation though.

Since the "universal grammar" hypothesis isn't really falsifiable, at best you can hope for some generalized equivalent that's isomorphic to the platonic representation hypothesis and claim that all human language is aligned in some given latent representation, and that our brains have been optimized to be able to work in this subspace. That's at least a testable assumption, by trying to reverse engineer the geometry of the space LLMs have learned.

I'm not actually comfortable saying that LLMs aren't conscious. I think there's a decent chance they could be in a very alien way.

I realize that this is a very weird and potentially scary claim for people to parse but you must understand how weird and scary consciousness is.

The problem is... that there is a whole amount of "smart" activities humans do without being conscious of it.

- Walking, riding a bike, or typing on a keyboard happen fluidly without conscious planning of each muscle movement.

- You can finish someone sentence or detect if a sentence is grammatically wrong, often without being able to explain the rule.

- When you enter a room, your brain rapidly identifies faces, furniture, and objects without you consciously thinking, “That is a table,” or “That is John.”

Is/was the same true for ASCII/Smalltalk/binary? They are all another way to translate language into something the computer "understands".

Perhaps the fact that it hasn't would lead some to question the validity of their claims. When a scientist makes a claim about how something works, it's expected that they prove it.

If the technology is as you say, show us.

  > The fact that we have figured out how to translate language into something a computer can "understand" should thrill linguists. 

But I think it is important to distinguish the ability to understand language and translate it. Enough that you yourself put quotes around "understanding". This can often be a challenge for many translators, not knowing how to properly translate something because of underlying context.

Our communication runs far deeper than the words we speak or write on a page. This is much of what linguistics is about, this depth. (Or at least that's what they've told me, since I'm not a linguist) This seems to be the distinction Chomsky is trying to make.

  > The main debate now is over the semantics of words like "understanding" and whether or not an LLM is conscious in the same way as a human being (it isn't).

My background is physics, and I moved into CS (degrees in both), working on ML. I see my peers at the top like Hinton[0] and Sutskever[1] making absurd claims. I call them absurd, because it is a mistake we've made over and over in the field of physics[2,3]. One of those lessons you learn again and again, because it is so easy to make the mistake. Hinton and Sutskever say that this is a feature, not a bug. Yet we know it is not enough to fit the data. Fitting the data allows you to make accurate, testable predictions. But it is not enough to model the underlying causal structure. Science has a long history demonstrating accurate predictions with incorrect models. Not just in the way of the Relativity of Wrong[4], but more directly. Did we forget that the Geocentric Model could still be used to make good predictions? Copernicus did not just face resistance from religious authorities, but also academics. The same is true for Galileo, Boltzmann, Einstein and many more. People didn't reject their claims because they were unreasonable. They rejected the claims because there were good reasons to. Just... not enough to make them right.

[0] https://www.reddit.com/r/singularity/comments/1dhlvzh/geoffr...

[1] https://www.youtube.com/watch?v=Yf1o0TQzry8&t=449s

[2] https://www.youtube.com/watch?v=hV41QEKiMlM

[3] Think about what Fermi said in order to understand the relevance of this link: https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness...

[4] https://hermiene.net/essays-trans/relativity_of_wrong.html

No, there is no understanding at all. Please don't confuse codifying with understanding or translation. LLMs don't understand their input, they simply act on it based on the way they are trained on it.

"And there's a fact here that's very hard to dispute, this method works. I can give a computer instructions and it "understands" them "

No, it really does not understand those instructions. It is at best what used to be called an "idiot savant". Mind you, people used to describe others like that - who is the idiot?

Ask your favoured LLM to write a programme in a less used language - ooh let's try VMware's PowerCLI (it's PowerShell so quite popular) and get it to do something useful. It wont because it can't but it will still spit out something. PowerCLI is not extant across Stackoverflow and co much but it is PS based so the LLMs will hallucinate madder than a hippie on a new super weed.

Also I used it for a few programming tasks I was pretty sure was in the datasets (how to draw charts with python and manipulate pandas frame). I know the domain, but wasn't in the mood to analyse the docs to get the implementation information. But the information I was seeking was just a few lines of sample code. In my experience, anything longer is pretty inconsistent and worthless explanations.

What you think about his argument about “not being able to distinguish possible language from impossible”?

And why is it inherent in ML design?

Does he assume that there could be such an instrument/algorithm that could do that with a certainty level higher than LLM/some ml model?

I mean, certainly they can be used to make a prediction/answer to this question, but he argues that this answer has no credibility? I mean, LLM is literally a model, ie probability distribution over what is language and what is not, what gives?

Current models are probably tuned more “strictly” to follow existing languages closely, ie that will say “no-no” to some yet-unknown language, but isn’t this improvable in theory?

Or is he arguing precisely that this “exterior” is not directly correlated with “internal processes and faculties” and cannot make such predictions in principle?

Are LLM's still the same black box as they were described as a couple years ago? Are their inner workings at least slightly better understood than in the past?

Running tens of thousands of chips crunching a bajillion numbers a second sounds fun, but that's not automatically "engineering". You can have the same chips crunching numbers with the same intensity just to run an algorithm to run a large prime number. Chips crunching numbers isn't automatically engineering IMO. More like a side effect of engineering? Or a tool you use to run the thing you built?

What happens when we build something that works, but we don't actually know how? We learn about it through trial and error, rather than foundational logic about the technology.

Sorta reminds me of the human brain, psychology, and how some people think psychology isn't science. The brain is a black box kind of like a LLM? Some people will think it's still science, others will have less respect.

This perspective might be off base. It's under the assumption that we all agree LLM's are a poorly understood black box and no one really knows how they truly work. I could be completely wrong on that, would love for someone else to weigh in.

Separately, I don't know the author, but agreed it reads more like a pop sci book. Although I only hope to write as coherently as that when I'm 96 y/o.

That's not a good argument. Neuroscience was constructed by (other) brains. The brain is trying to explain itself.

> The truth is that all of the progress on machine learning is absolutely science.

But not much if you're interested in finding out how our brain works, or how language works. One of the interesting outcomes of LLMs is that there apparently is a way to represent complex ideas and their linguistic connection in a (rather large) unstructured state, but it comes without thorough explanation or relation to the human brain.

> Chomsky is [...] pretty flippant about what this stuff can do.

True, that's his style, being belligerently verbose, but others have been pretty much fawning and drooling over a stochastic parrot with a very good memory, mostly with dollar signs in their eyes.

To be fair the article is from two years ago, which when talking about LLMs in this age arguably does count as "old", maybe even "really old".

I've been saying this my whole life, glad it's finally catching on

I remember having thoughts like this until I listened to him talk on a podcast for 3 hours about chatGPT.

What was most obvious is Chomsky really knows linguistics and I don't.

"What Kind of Creatures Are We?" is good place to start.

We should take having Chomsky still around to comment on LLMs as one of the greatest intellectual gifts.

Much before listening to his thoughts on LLMs was me projecting my disdain for his politics.

It is not science, which is the study of the natural world. You are using the word "science" as an honorific, meaning something like "useful technical work that I think is impressive".

The reason you are so confused is that you can't distinguish studying the natural world from engineering.

Because languages have many similar concepts so the operator inside the Chinese room can understand nearly all the concepts without speaking Chinese.

And the LLM can translate to and from any language trivially, the inner layers do the actual understanding of concepts.

The analogy I've used before is a bright first-grader named Johnny. Johnny stumbles across a high school algebra book. Unless Johnny's last name is von Neumann, he isn't going to get anything out of that book. An LLM will.

So much for the Chinese Room.

The compression we use in languages to not label impossible adjectives against impossible nouns (green ideas is impossible as ideas don't have colors, we could have a suffix on every noun to mark what can be colored and what cannot) is because we need to transfer these over the air, and quickly, before the lion jumps on the hunter. It's one of the many attributes of "languages in the wild" (Chinese doesn't use "tenses" really, can you imagine the compressive value?), and that's what Chomsky says here:

Proceeding further with normal science, we find that the internal processes and elements of the language cannot be detected by inspection of observed phenomena. Often these elements do not even appear in speech (or writing), though their effects, often subtle, can be detected. That is yet another reason why restriction to observed phenomena, as in LLM approaches, sharply limits understanding of the internal processes that are the core objects of inquiry into the nature of language, its acquisition and use. But that is not relevant if concern for science and understanding have been abandoned in favor of other goals.

Understand what he means: you can read a million text through a machine, it will never infer why we don't label adjective and nouns to prevent confusion and "green ideas". But for us it's painfully obvious, we don't have time when we speak to do all that. And I come from a language when we label every noun with a gender, I can see how stupid and painful it is to grasp for foreigners: it doesn't make any sense. Why do we do it ? Ask ChatGPT, will it tell you that it's because we like how beautiful it all sounds, which is the stupid reason why we do that ?

My understanding is that context sensitive grammars _can_ allow for recursive structures that are beyond cfgs, which is precisely why they sit below csgs in terms of computational complexity.

I don’t agree or disagree that LLMs might be, or are, capable of parsing (i.e., perception in Chomsky’s terms, or, arguably, “understanding” in any sense). But that they can learn the grammar of a “complex cfg” isn’t a convincing argument for the reasons you indicate.

I don’t believe for one second that LLMs reason, understand, know, anything.

There are plenty of times LLMs fail to generate correct sentences, and plenty of times they fail to generate correct words.

Around the time ChatGPT rolled out web search inside actions, you’d get really funky stuff back and watch other code clearly try to catch the run away.

o3 can be hot garbage if you ask it expand a specific point inside a 3 paragraph memo, the reasoning models perform very, very poorly when they are not summarizing.

There are times where the thing works like magic, other times, asking it to write me a PowerShell script that gets users by first and last name has it inventing commands that flags that don’t exist.

If the model ‘understood’, ‘followed, some sort of structure outside parroting stuff it already knows about it would be easy to spot and guide it via prompts. That is not the case even with the most advanced models today.

It’s clear that LLMs work best at specific small tasks that have a well established pattern defined in a strict language or api.

I’ve broken o3 trying to have it lift working python code, into formal python code, how? The person that wrote the code didn’t exactly code it how a developer would code a program. 140 lines of basic grab some data generate a table broke the AI and it had the ‘informal’ solution in the prompt. So no there is zero chance LMMs do more than predict.

And to be clear, it one shot a whole thing for me last night, using the GitHub/Codex/agent thing in VS code, probably saved me 30 minutes but god forbid you start from a bad / edge / poorly structured thing that doesn’t fit the mould.