Is 'Attention' the Essence of Intelligence?(quantamagazine.org) |
Is 'Attention' the Essence of Intelligence?(quantamagazine.org) |
Most of the semantics of a sentence are indicated by the precise order the tokens appear. This is easiest to see in a language like English, since it is visible in two very prominent ways.
First, the morphology is fairly derivational, meaning you can keep adding suffixes to things and English speakers will generally get what you mean. For example, add some more suffixes to that word: derivational-ish-ly. You've probably never heard this specific variant, but you still probably have some idea of what it means anyway. The transformer can only capture these semantics if it has some representation of the fact that the suffixes inflect the meaning of the word's root (presumably something like `deriv`). English encodes this entirely by the position of the suffixes relative to the root. Without the positional information, there is no way to recover it, and sentences would lose much of their meaning, since English depends heavily on this construction.
Second, English has relatively few cases, and thus relies very heavily on the order of words to indicate what is the subject and what is the object. English specifically is Subject-Verb-Object, and the instinct of native English speakers to use SVO ordering is so strong, you will find them inserting nouns and pronouns into sentences that do not actually have subjects at all. "It's raining outside" is the most famous example. This sentence has no subject at all, and yet native English speakers feel a strong need to place "it's" at the beginning.
If the transformer did not have positional encodings it would be completely hopeless to recover what is the subject and what is the object from English sentences. There simply is no other way to reliably tell, other than inspecting the word positions.
It's worth noting we also do have the counterfactual. Generative language models that explicitly exclude positional information, like Latent Dirichlet Allocation, are only able to (essentially) cluster word co-occurrences. That is not as useless as it sounds—if I say "fire house burning ambulence" you kind of get the point even without a subject or an object—but it certainly does not generate the lucid, provoking sentences that modern LLMs do.
German: es regnet French: Il pluet
I don't see why English is special here.
French is descended from Latin, which is "pro-drop" in the sense that you'd drop the "dummy subject" and simply write "pluit." Later, Old French, like English, lost most of its cases. Because of that, many word endings started sounding very similar (a problem known as "agreement syncretization"). The pro-drop "null" subject appears in the record roughly as that happens, presumably to help disambiguate.
This is an entirely different path than English took, and it's also not a path any other Romance language took. Spanish isn't even subject-prominent. It fully allows the "null" construction, e.g., "llueve", which literally is just "rains."
One word (rains) is the full sentence. Because nothing else needs to be said.
The rain is not magical, the environment is producing it. The environment is raining. It's raining outside means the environment is producing rain outside.
Who could possibly confuse that as lacking a subject.
https://www.cambridge.org/core/journals/journal-of-linguisti...
I guess the author means in contrast to some cases like "Pluvas" in Esperanto, see https://www.youtube.com/watch?v=fOBkKcbJUAE for a nice employ of it.
Italian, Spanish and Latin (for example) can generally avoid to express explicitly the subject in a dedicated pronoun, as the verb already encode the it through specific flection. In contrast in French and German only in imperative tense one can formally make such an ellipsis (colloquially of course one can encounter more nuances).
So nothing exceptionnal
But these days, there are many more options. They all maintain this positional information in one way or another. The one you cite (as I understand it) drops the positional encodings after pretraining, and keeps (e.g.) the causal mask.
I would rather say it’s derived from contextual element. And in linguistic context which encodes more interpretation cues, order of morphemes drop symmetrically in semantical importance, though it can still affect interpretation on pragmatic level. Latin for example already offers more flexibility regarding word order, though it’s used mostly in poetry/stylised-rhetoric. And as German show having the case encoded in the nouns don’t even prevent mandatory rigidity in sentence structure.
There isn't really a way to get around this. Natural language IS structured as a recursive tree. Our job is to extract this tree from a flat stream of tokens (or similar), and straightforwardly the only way to do that is to make the ordering of the tokens meaningful. There is no other option! This is how 100% of parsing works in any context whatsoever, both theoretically and practically.
And it turns out you probably need massive compute, about 1/3 RLHF, all the worlds data and millions of people connecting their private data to the AI to learn from amongst other things :-/
Of course, attention is almost certainly a heuristic readers use, but to even bring it up as key to intelligence begs so many questions about LLMs and the mind...
https://www.quantamagazine.org/is-attention-the-essence-of-i...
"Sentences with pronouns such as “it” in them illustrate the challenge that attention overcomes. Consider: “The animal didn’t cross the street because it was too tired.” And: “The animal didn’t cross the street because it was too wide.” To translate these sentences correctly into French, an AI model must determine that, in the first one, “it” refers to the animal, which is a masculine word in French, so the translation should use that form of the pronoun, il. But in the second sentence, “it” refers to the street, a feminine word that requires a different pronoun, elle. How can the machine figure this out?"
"Bahdanau remembered doing translation exercises when he was learning English in school. To write each next word of his translated sentence, he’d have to glance all around the source text to figure out what should come next. “Literally you have your source sentence on the left and your writing on the right, and you’ll be, like, turning your head, left-right, left-right,” he said. “That process is what I expressed as a neural mechanism.”
That simplicity seems to me the real tell that this is the real insight. The author does a good job of walking the ready past a long list of distractors and conjuring a sense of a path to an insight, and then, there's the insight - a simple encoding of a simple task: "What is the next word? How do I figure that out? Left-right. Left-right"
"Spreading activation" would be a better analogy.
----
It's mostly harmless, though it occasionally results in head-scratchers from the biological POV. I once saw a ML paper talking about "retinal attention", which is funny, because the retina is one of the few places in the central nervous system where afawk, attention is impossible.
There's simply no efferent (brain->retina) non-motor nerves capable of biasing photoreceptor output; the retina is largely ballistic. (Eyeball motion is not attention.)
How can we still have it so backwards. Language is just a medium to convey information on an underlying reality. LLM do not model the medium, they model the underlying reality (so far as the medium allows it).
But then dumbly concludes : oh is language intelligence ? ("Perhaps our form of intelligence is tantamount to language modeling, too?"). C'mon.
Imagine you have a series of (x,y) coordinates. You receive a new x coordinate, and you'd like to predict the corresponding y. So, the absolute dumbest thing you can do is find the nearest x, and then report its corresponding y as your prediction. The second dumbest thing you can do is find the n closest x coordinates, and then average their y values together, and output that as your predicted value. And the third-dumbest thing you can do is that n-closest-average thing, but give the "closest" x coordinates the most "weight" when you average them. (Technically, not always "closest": you have some "kernel" function which does the weighting, but "closeness" is a good kernel so I will use it here.)
That third-dumbest thing is Nadaraya-Watson kernel smoothing. And that's what attention does. You have a big list of vectors. You compute a weighted average of those vectors. You now have a new vector. You can do what you want with it, but LLMs will generally turn that into a distribution over token types, from which you'll generally want to select either the top one, or weighted-randomly one from the top k (known as "increasing the temperature").
With all that said, I'm not sure "Kernel smoothing is all you need" is actually materially better than "attention is all you need" for 99.999% of humans. :)
https://scikit-learn.org/stable/modules/generated/sklearn.im...
https://scikit-learn.org/stable/modules/generated/sklearn.me...
I don't have any specific experience in ML but I always liked the formalism presented in https://www.youtube.com/watch?v=J1YCdVogd14 as a way to unify everything
Now if you could please explain the rest of math in a similar manner?
I thought both were useful in their own way, it's not like one replaced the other right?
--
A computer can't read a file, it is just sending electric signal, sensing the voltage, probing the magnetic field, etc.
"read" a file is anthropomorphized terms for what the machine do when processing a file.
--
Maybe we should just stop worrying and embrace the new meaning of those words?
As a former perceptual neuroscientist, I can assure you that ML "attention" has very little to do with biological attention.
People are doing what people do: they're desperately moving the goal post to keep pretending humanity is particularly special and that intelligence is particularly special. Unique snowflakes and all that, we're the center of the universe. It's nothing more than elaborate human weakness & fragility on display.
We're in the first inning of this, intelligence is not going to end up being special, it's going to be trivial and low cost. See: Chinese models ten years from now.
Astra, Opus, Fable, Argon - they're all radically beyond the top 1% of humanity intellectually already, and it's going to get a lot worse quickly. The sole aspect the top 1% beat them on, and increasingly rarely, is at hyper specialization, where a person is elite at what they do. For $20-$100-$200 per month I can get access to an omnimath genius intellect far beyond any human that has ever existed, and that AI will work for me like a dog. Humanity should be scared, and the reason HN hates AI so much, is because it will wipe out very large numbers of their lucrative jobs that they imagined would form long-term careers (six figures, stock options, path to wealth, elitism re competition).
And suddenly we're looking at people who read ticker tape or by following with their finger for insights into neuroscience?
I thought with NoPE schemes you don't even need it during pretraining
https://arxiv.org/abs/2305.19466
https://sebastianraschka.com/llm-architecture-gallery/nope/
Given the causal mask and the way attention works LLMs can implicitly propagate a signal for ordering
We survived comparisons to hydraulics, clockwork, steam, circuits, information processing, and networks. I'm positive we'll survive comparisons to attention and models too.
If it's an argument from pedantism, let us remind ourselves to be language descriptivist than language prescriptivists. Prescriptivists never win.
So instead of being a useful, if imperfect, analogy from an outside domain, trying to use ML attention to understand biological attention just results in confusion (like this article).
Neuroscientists aren't going to tell ML researchers they can't use "attention", but nor are they going to pay attention to them (pun intended). Neural networks are almost always too divergent from biology to shed light on real neurons.