Not suggesting that I don't say stuff on autopilot sometimes but for many people, it's their only mode of operation. They never actually think about anything from first principles. Their whole approach to language is just chaining catchphrases together. It's how a toddler thinks; it seems like many people never moved past that stage of development.

> Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot.

I think this metaphor is so strained as to not be useful. I think key here is that the authors say "without any reference to meaning", which is a heavily loaded term, that does definitely apply to parrots, but does not apply when you apply it to immense bodies of text.

Namely that language embeds meaning in language. A sentence being written by a human (as a starting point) is designed to have consistent meaning. While it is possible to write syntactically correct meaningless text, that is not what most of human language has done; the meaning cannot be removed from the text.

This I think is clarifying, from the same paragraph in the text:

> ... the training data never included sharing thoughts with a listener, nor does the machine have the ability to do that.

That's just facially incorrect. The training data is entirely about sharing thoughts with a listener. Else why is the text being written?

> Meanwhile, O, a hyper-intelligent deep-sea octopus who is unable to visit or observe the two islands, discovers a way to tap into the underwater cable and listen in on A and B’s conversations. O knows nothing about English initially, but is very good at detecting statistical patterns. Over time, O learns to predict with great accuracy how B will respond to each of A’s utterances. O also observes that certain words tend to occur in similar contexts, and perhaps learns to generalize across lexical patterns by hypothesizing that they can be used somewhat interchangeably. Nonetheless, Ohas never observed these objects, and thus would not be able to pick out the referent of a word when presented with a set of (physical) alternatives.

This seems kind of obviously wrong at least in the context of coding agents. These models get trained on actual output of the previous version of the model doing its job, often "IRL" on a real computer/project. It's like O is in the conversation for years now and learning from his own interactions between A <-> O <-> B, where A is the human and B is the computer.

The idea O ontologically has never "observed" "these objects" or referents is philosophically strained. Have I observed the moon, or a finger pointing at the moon? Have I observed `sed` more than Fable?

I don't understand what we're setting the record straight on. This is the core point of dispute, and the author just blazes past it to focus on other things. I'm glad to hear "stochastic parrot" isn't intended as an insult, and I agree that it's not right to think of LLMs as a box with a little homunculus inside replying to you. But to me it seems obvious that LLMs are not repeating patterns without comprehension and do understand what they are saying; otherwise they would not be capable of doing things they routinely do.

...

> It knew that this is a dangerous thing I should not do in real life

From the ChatGPT response you linked, all I see for sure is some matches on the following patterns:

  drop $thing from skyscraper --> bad behavior
  drop $thing --> physics
  can of $stuff --> contents in/out of can

It also does an incredibly poor job of answering your question. It makes no attempt to explain what might actually happen. If it has been trained on the entire corpus of medical science, and it is indeed intelligent, then surely it can reference ballistics studies and give you a very detailed and thorough theory of what--exactly--injuries you might expect from a 12oz can being dropped from the height of a skyscraper. Calculating the terminal velocity and therefore the momentum of the can is trivial. Characterizing the physics of the impact on various parts of a human body is trivial. If it actually understood your request why didn't it just answer the question?

It's a rhetorical question. LLMs do not "understand". It is completely outside their capability. "Understanding" is something we impose upon their output (to loosely quote TFA). [edit] I think the most powerful evidence for a lack of any understanding whatsoever is all the stuff about the cherries being in or out of the can. Yes, cans contain things. That is not a profound observation, nor is it at all relevant to the question. If you drop an empty can off a skyscraper nothing meaningful will happen. And, no, probably dumping all the cherries out won't hurt anyone or cause a slipping hazard... It's also not particularly relevant to point out that dropping things off skyscrapers is bad behavior. But that's more forgivable from a CYA standpoint.

I believe you are projecting something that is not there onto a completely mindless stochastic process.

[another edit] I found this helpful, you may also: https://www.0xkato.xyz/how-llms-actually-work/

I disagree that decoupling "knowing" from "understanding" is well motivated or useful in any way.

> I don't think that it is useful to say that LLMs understand anything they say, or that we say to them.

This is starting to seem like a Chinese Room disagreement, so I'll just say that what that thought experiment demonstrated is that there is no robust or meaningful way to distinguish semantics from syntactic processing given everything we know about reality, and that LLMs were a fantastic demonstration of this illusion, eg. that "semantics" are reducible to a relational network of syntactic elements.

Now if they had said, "Imagine your average American ..." (/s)

Cf. https://machinelearning.apple.com/research/illusion-of-think...

That's what Bender doesn't get. Some folks tried techniques different from her favorite techniques and made unbelievably fast progress across a wide range of previously insoluble problems (regardless of whether they satisfy properties Bender believes are required for intelligent systems).

I think it's safe to say that none of the main LLMs have some sort of self-awareness as we think of it in humans, but I also expect that more sophisticated systems in the future could. If I had to guess, they would have significantly more activity going on in the network- not just individual end-to-end top-down forward graph, along with cycles instead of trees, and the neurons themselves would be sigifnificantly more capable (effectively little state machines that run functions on input that passes through). I guess also you'd want to have some sort of rules-based (but statistically trained) execution component managing everything.

The point of the paper, in fact, is that language models are getting "too big", and another approach is needed to make progress, so they were certainly predicting things about later models.

With that said, they talked about "pure" language models, so it is fair to say that they didn't talk about, say, LLMs that are multimodal or that have tool use, which are advances that happened after their paper.

It's her own paper, she's citing herself. And the full sentence is "As we discuss in §5, LMs are not performing..." so she's actually just teeing up the same claim she's made previously for further discussion. Why are you trying to claim I'm misrepresenting her words?

> did you read the article after that line or was there a shortage of attention span??

I did. It's more of the same, so there's nothing to say about it.

For example, note my use of the phrase “turned out” and then consider whether your main point about the release date of plus 4.8 vs the paper makes any sense at all?

> how do you know what is novel?

Because I work on novel ASICs that were created by my company and have never been used or seen outside of my company?

But the different terms imply different mental models of what LLMs are and can do. If you take two people, one who thinks of them as "artificial intelligence" and one as "stochastic parrots" (with all the implicit context and connotations of the individual words composing them), what mental model would have led to better predictions of LLMs' future circa 2020?

The "stochastic parrots" phrase is very dangerous in that frame. People read far more into what capabilities it implies are (im)possible than the narrow technical description the authors originally argued for. If all they are is spicy autocomplete or pastiche plagiarizers, there's nothing serious to worry about. And when an opposition gets stuck in a trough that mindlessly dismisses their future capabilities out of hand because of a bad mental model, it renders them ineffective at preventing the worst outcomes.

It doesn't make sense to say a model is in possession of its self. That's exactly the sort of poetic anthropomorphization that Bender was criticising here, and a good reason to not refer to an LLM as "an AI".

The math is based upon theories of how the brain work, but even if those theories are right, this math is a great simplification and subset of what organic brains do.

It's hard to imagine how high intelligence could be selected for without social component that provides the individuals a lot of pressure to outsmart each other.

The claim is odd in another way: you can train a person on non-working code and they'll produce nonsense. That doesn't mean people are just stitching together words they've previously seen.

And, not to insult you, but it's quite obviously wrong. As a mental model it fails to explain basic capabilities. How can an LLM follow elaborate instructions? How can it respond appropriately to user input, when the user input doesn't match any previously seen text? Hell - how does it even balance parentheses? There is no way to explain any of this without conceding that the LLM has semantic understanding. It knows that this comes after that, but "this" and "that" can be at an arbitrary level of abstraction.

Sure - they generate text "like" text they've seen before. That "like" does a ton of heavy lifting.

Exhibit A.

I will change my mind if someone demonstrates such a robot. Absent this demonstration, cockroach-level AI is still an unsolved problem. Given how ignorant and arrogant and wealthy AI researchers are, it will remain unsolved. I don't think anyone alive today will live to see a robot smarter than an ant.

"This sentence has five words" is going to appear far more often than "This sentence has four words". This is the entire premise of LLMs working at all, stochastic parrots or otherwise.

Isn't that pattern matching essentially?

The whole brain is most definitely not dedicated to prediction, and I don't think "prediction" is a very useful model anyways. You could say the hippocampus is for "prediction" if you really squint, but that's underselling what it's doing. And the basal ganglia operates off of prediction error, but it's more about regulating short timescale feedback loop learning than learning itself - speaking somewhat loosely because it's a particularly ancient structure in the brain and things get muddied.

LLMs obviously don't have functional equivalents to either of those - sure they can remember facts, but they can't intake new facts the way we can, the hippocampus is set up completely differently. But higher reasoning is humans is the neocortex, and that LLMs can do, and there's even structural parallels - LLMs and the neocortex are both implemented in layers, and you can even (with a shit ton of analysis) watch how LLMs walk the geometry across layers. The neocortex needs a lot fewer layers than LLMs, but that's because it's not pure feedforward across layers; the neocortex already does what the "looped LLM" people are trying to do.

> Similarly, if you agree with the Manifold Hypothesis, then all machine learning models operate on manifolds. I agree it's an exciting thought, but then I don't know what would distinguish an LLM from a VAE or SVM in terms of operating over a low-dimensional manifold embedded in high dimensional spaces - maybe just scale?

I view things more mathematically, and in math and physics it frequently turns out that there's really only one solution to a problem, or correct way to model something - and then everything else is just isomorphisms (which may be interesting objects in their own right!).

My interpretation/guess (informed by research, but with a lot of still unanswered questions) is that the manifold is fundamentally the structure that emerges from natural language: natural language has a computational model, just like Church/Turing machines have a computational model. Reimannian manifold is also underselling it a bit, it's also Finsler, and RoPE adds additional structure to that manifold (and when we get to RoPE it wouldn't surprise me if the neocortex and LLMs do diverge significantly in how they compute, but does the structure change? dunno yet).

Basically, I'm fairly confident in what I say about machine learning models that understand natural language; for non-NL machine learning, maybe the same structure emerges with enough complexity, but I haven't really pondered that.

The evidence is literally every other evolved form of intelligence. Including, despite your speculation, octopuses. Do recall "the environment" also includes other intelligent agents an animal needs to compete with. Does that help you understand? A runaway process might be required for human level intelligence but clearly not in general.

Basically all octopuses are solitary and die after breeding. LPSOs are nested in one group of octopuses. You're proposing that one group of octopuses developed social behavior and multiple breeding, then had a bunch of descendants who went back to exactly the normal octopus behavior. Hopefully it's obvious why this fails Occam's razor.

That's not what I said. What I said was that it's physics that provides the ground truth.

>you could, at least in theory, falsify the entire experience of the bird

It wouldn't be a bird anymore, but a dysfunctional cyborg with false perceptions.

>There is nothing in a bird's brain that directly percieves reality.

Yes, of course there is. Animal sensory organs do not produce false information, nor do they provide the brain an interpretation of what they perceive. The brain may fail to distinguish hallucinations from reality, but those are processes internal to the brain. What it gets from the body is raw physical measurements, and what it sends out is raw motor commands.

A multimodal model doesn't have the same direct access to reality, it just has collections of words and images. It has no capacity to determine the reality of a photograph of a sunset or a CGI render of a dragon. The word "real" is itself meaningless to it; they're both real in that they appear in its training corpus. It lacks the capacity to investigate these stimuli in any way, and can just learn to associate different stimuli in arbitrary ways that appear to make sense to us, but nothing else.

I don't know how or if I will notice. That's the biology, chemistry and physics of the brain that I don't know. I hope someone is looking into it. But this does not mean in any way that LLMs are similar to our brains!!!!!! WE DON'T KNOW HOW OUR BRAINS WORK. So going back to language modeling - language models were stochastic in nature when that paper was written, they still are albeit we are trying to make them as deterministic as possible.

Ok. Now. I think you're adding something to the description above. Maybe what you're describing is something "emergent," or maybe it's basically just word vectors that were built in on purpose. You may be adding something correct, or something incorrect. Fine.

But it's not reasonable to say that the "reductive" description above is a "lie". It's not. It's more like a recipe. If you look at correct instructions for making steak, and you call the author a "liar" then you are missing something important.

You seem to believe, on a more fundamental level, that LLMs are simply not capable of producing text that has deeper connections to itself or represents abstract thoughts. In my opinion, 99% of text written by humans does not show this, just as 99% of text produced by LLMs does not show this, but both have the capability, and I don't believe that LLMs are constrained in such a way that they can never do this.

And this is a reply to her comment about water usage where it becomes clear she's not arguing in good faith. https://x.com/AndyMasley/status/1990498830131888173

And basically the burden of proof is on your side. As often is the case, discussions on AI start to resemble discussions centred on religion and belief in god. Your only line left here is “yes but you cannot prove LLM’s are conscious” - which I can’t, since proving that Unicorns do not exist is pretty senseless activity.

So basically I’d say LLM’s are stochastic parrots (or whatever term you want to use) until we can very definitely prove that they indeed think and have some sort of consciousness.

Even amongst municipal water usage, it seems hard to figure out how much DC usage factors in to other uses because often they’re not required, and therefore don’t, disclose their usage.

All I’m saying is that it’s not a closed case and not everyone complaining about DC water usage is a Russian propaganda bot.

Meanwhile, datacenters sacrifice water by sending it to Mars in order to compute criminal deepfakes of innocent politicians.

So, on the matter of scale: there likely isn't a cap on water use of these datacenters. Both the heat emission and usage levels for these systems will likely go up unless there is a fundamental technical breakthrough.

On the matter of utility: As a sibling of GP mentioned, the utility of food is clear.

On the matter of polution: I am not remotely read on waste water and contamination due to industrial agriculture. Is this also something where the judgmental scale is tipped in favor of food production vs cooling systems?

> Claim that a data center is using 1000x as much water as a city of 88,000 people, where it’s actually using about 0.22x as much water as the city, and only 3% of the municipal water system the city relies on. She’s off by a factor of 4500. This is the single largest error in any popular book that I’ve found on my own, and to my knowledge I’m the first person to notice it.

Yeah, she might be wrong. But a data center also using 1/5th of the water consumption of an 88k person city should still be what are debating. We also have a base rate fallacy, we don't know know how well or how poorly they are using water. Nearly all criticisms of AI data center criticisms boil (no pun intended) down to yeah, but what about almonds or rice or xyz. That isn't a healthy way to adjust how we talk about data center water consumption.

It is a classic debate tactic. Someone makes an argument, then buttresses that argument with a number. You attack the number, pretending that you didn't just correct the number, but also invalidate the original argument. We shouldn't be using these tactics to talk about the tragedy of the commons.

I worked for a hyperscaler, I poked around a bit about water usage both internally and externally and it wasn't good. There was little to no thought other than, "we can pay X to have water delivered, doesn't matter if it sourced responsibly." (to glibly paraphrase, company policy is to never write the honest part down)

Look at how hard Google fought to not have water data released in The Dalles Oregon for their DC there. Many DCs are supplied by water that is meant for humans, sourced from aquifers that took hundreds of thousands of years to fill, that are being depleted faster than they are filled, already.

I think AI is powerful tool, but we still can't give DC expansion a pass.