You could have invented transformers from memory(alexwang.ai) |
You could have invented transformers from memory(alexwang.ai) |
For an LLM, the keys are the tokens in your context, and the values are … what exactly?
Q = XW_q
K = XW_k
V = XW_v
From there, you can think of K and V like keys and values in a Python dictionary, and Q as the key you pass (e.g. d[Q] in Python to return some value V). The difference in an attention layer is that the query is matched to each key using a dot product, and rather than getting a single value back you get a weighted sum of values.
I recall awhile back hearing someone ask whether intelligence was actually the same or similar to “compression” or storing all of this knowledge in weights - IE the ability to create something from reason or weights or relationships / connections. This article makes it even more interesting!
If you think that is crazy, ohh boy, just wait til you see all the other papers underlying LLMs, AI, and optimization in general.
I appreciate this post because starting from linear algebra is the correct way to form an intuition of the higher dimensional semantic spaces we're dealing with.
NNCP: Lossless Data Compression with Neural Networks (2019)
1. It uses math notation. This is a software development site, wouldn't this be much more clear to this audience written as psuedo code instead of a complex series of greek and latin letters?
2. It doesn't step through a concrete example. What is hollow R raised to the (M * d_v) ? Is it a number, a set, a list, a matrix?
To use an example I understand: RAG used to seem complicated to me when only math people were talking about it in math words. For example, let's take a line from wikipedia on embeddings:
> the representation is a real-valued vector that encodes the meaning of the word in such a way that the words that are closer in the vector space are expected to be similar in meaning
What made it clear was learning that generating an embedding is just a function(string) -> list[float], and RAG is just taking a list[float] for your prompt and finding other list[float] that are close to it in your dataset.
So what would make this article more clear, is step through a simplified example of putting a value into a memory store and querying that memory store, showing the actual data structures.
At the risk of stating the obvious, this is a forum where people can post arbitrary links to other sites, some of which might have audiences that might not be "the exact set of people who read Hacker News". I'm not sure you'd be able to make a compelling argument that alexwang.ai is aimed at software developers and no one else.
This article is a bit more of a grey area. It is on a subject that is deeply relevant to this site and I'm sure some HN readers will get value from it. My point is that an even better submission to HN would have been an article on the same topic but written for programmers rather than mathematicians.
This is pretty approachable maths, and explain really well. I've seen far worse in programming language theory papers, which would qualify as "software development" in my mind. There are plenty of users of this forum that would find this paper useful and interesting, with sufficient math background to read it to boot.
>It's easy to forget that the average person probably only knows the formulas for Olivine and one or two Feldspars
That being said the llm is indeed the universal translator — the babelfish supreme — and is excellent at translating whatever you want into pseudocode in your language of choice.
Perhaps, but HN apparently is pretty hostile to people explaining why something is not clear and how it could be more clear based on all the downvotes. I'll keep that in mind: "If you don't understand something, shut up!"
Also not an issue: the quality of the article.
Expect to be challenged by content here. Everybody is from time to time. Asking for a mathy analysis to drop the math, fundamentally misunderstands the value of the math.
Clear explanations are a good thing, but different explanations help people in different ways. Your explanation seems helpful to me, and reminds me of Ben Affleck's explanation in the GQ interview. But Ben Affleck connected it directly to arrays of RGB pixel data, which clicks more with me personally, but probably wouldn't with everyone. It certainly didn't make things any clearer to the interviewer.
Imagine a nursing home that serves steak every day. One resident says "this food isn't helpful to me because I no longer have teeth to chew it. Wouldn't this be better for everyone if we had soup?". They're not suggesting outlawing steak, after all some people will get value from it. They suggesting a way to better serve the entire population of the nursing home.
HN is a site for programmers (or specifically "hackers"). Posting an article to a site for programmers in math notation is like posting a French article to an English message board. Some people will be able to read it with varying levels of difficulty, but wouldn't it be better to post an English version of the same information so everyone can read it easily?
https://news.ycombinator.com/hackernews.html
"... the thing that makes hackers such good startup founders is that they care about more than business. They have intellectual curiosity driving them as well as the desire to make money. So the way to make a news site for startup founders is to make it be about more than just startups. ... If you had to reduce it to a sentence, the answer might be: anything that gratifies one's intellectual curiosity."
My thought was to show some grace, considering that not everyone's been around here for 20 years or knows that "hacker" doesn't apply only to computer programmers. That was my thinking anyway.