Entropy of a Large Language Model output(nikkin.dev) |
Entropy of a Large Language Model output(nikkin.dev) |
The entropy of ChatGPT (as well as all other generative models which have been 'tuned' using RLHF, instruction-tuning, DPO, etc) is so low because it is not predicting "most likely tokens" or doing compression. A LLM like ChatGPT has been turned into an RL agent which seeks to maximize reward by taking the optimal action. It is, ultimately, predicting what will manipulate the imaginary human rater into giving it a high reward.
So the logits aren't telling you anything like 'what is the probability in a random sample of Internet text of the next token', but are closer to a Bellman value function, expressing the model's belief as to what would be the net reward from picking each possible BPE as an 'action' and then continuing to pick the optimal BPE after that (ie. following its policy until the episode terminates). Because there is usually 1 best action, it tries to put the largest value on that action, and assign very small values to the rest (no matter how plausible each of them might be if you were looking at random Internet text). This reduction in entropy is a standard RL effect as agents switch from exploration to exploitation: there is no benefit to taking anything less than the single best action, so you don't want to risk taking any others.
This is also why completions are so boring and Boltzmann temperature stops mattering and more complex sampling strategies like best-of-N don't work so well: the greedy logit-maximizing removes information about interesting alternative strategies, so you wind up with massive redundancy and your net 'likelihood' also no longer tells you anything about the likelihood.
And note that because there is now so much LLM text on the Internet, this feeds back into future LLMs too, which will have flattened logits simply because it is now quite likely that they are predicting outputs from LLMs which had flattened logits. (Plus, of course, data labelers like Scale can fail at quality control and their labelers cheat and just dump in ChatGPT answers to make money.) So you'll observe future 'base' models which have more flattened logits too...
I've wondered if to recover true base model capabilities and get logits that actually meaningful predict or encode 'dark knowledge', rather than optimize for a lowest-common-denominator rater reward, you'll have to start dumping in random Internet text samples to get the model 'out of assistant mode'.
Of course humans employ different thinking modes too - no harm in thinking like a stone cold programmer when you are programming, as long as you don't do it all the time.
It is indeed alarming that the future 'base' models would start with more flattened logits as the de-facto. I personally believe that once this enshittification is recognised widely (could already be the case, but not recognized) then the training data being more "original" will become more important. And the cycle repeats! Or I wonder if there is a better post-training method that would still withhold the "creativity"?
Thanks for the RLHF explanation in terms of BPE. Definitely easier to grasp the concept this way!
This isn't strictly true. It is still predicting "most likely tokens"! It's just predicting the "most likely tokens" generated in a specific step in a conversation game; where that step was, in the training dataset, taken by an agent tuned to maximize reward. For that conversation step, the model is trying to predict what such an agent would say, as that is what should come next in the conversation.
I know this sounds like semantics/splitting hairs, but it has real implications for what RLHF/instruction-following models will do when not bound to what one might call their "Environment of Evolutionary Adaptedness."
If you unshackle any instruction-following model from the logit bias pass that prevents it from generating end-of-conversation-step tokens/sequences, then it will almost always finish inferring the "AI agent says" conversation step, and move on to inferring the following "human says" conversation step. (Even older instruction-following models that were trained only on single-shot prompt/response pairs rather than multi-turn conversations, will still do this if they are allowed to proceed past the End-of-Sequence token, due to how training data is packed into the context in most training frameworks.)
And when it does move onto predicting the "human says" conversation step, it won't be optimizing for reward (i.e. it won't be trying to come up with an ideal thing for the human say to "set up" a perfect response to earn it maximum good-boy points); rather, it will just be predicting what a human would say, just as its ancestor text-completion base-model would.
(This would even happen with ChatGPT and other high-level chat-API agents. However, such chat-API agents are stuck talking to you through a business layer that expects to interact with the model through a certain trained-in ABI; so turning off the logit bias — if that was a knob they let you turn — would just cause the business layer to throw exceptions due to malformed JSON / state-machine sequence errors. If you could interact with those same models through lower-level text-completion APIs, you'd see this result.)
For similar reasons, these instruction-following models always expect a "human says" step to come first in the conversation message stream; so you can also (again, through a text-completion API) just leave the "human says" conversation step open/unfinished, and the model will happily infer what "the rest" of the human's prompt should be, without any sign of instruction-following.
In other words, the model still knows how to be a fully-general, high-entropy(!) text-completion model. It just also knows how to play a specific word game of "ape the way an agent trained to do X responds to prompts" — where playing that game involves rules that lower the entropy ceiling.
This is exactly the same as how image models can be prompted to draw in the style of a specific artist. To an LLM, the RLHF agent it has been fed a training corpus of, is a specific artist it's learned to ape the style of, when and only when it thinks that such a style should apply to some sub-sequence of the output.
Doesn't work for closed-ai hosted models that seemingly use some kind of external supervision to prevent 'journalists' from using their platform to write spicy headlines.
Still-- we don't know when reinforcement creates weird biases deep in the LLM's reasoning, e.g. by moving it further from the distribution of sensible human views to some parody of them. It's better to use models with less opinionated fine tuning.
How is this not deterministic? Randomness is intentionally added via temperature.
But I see the same misconceptions as always around “hallucinations”. Incorrect output is just incorrect output. There is no difference in the function of the model, no malfunction. It is working exactly as it does for “correct “ answers. This is what makes the issue of incorrect output intractable.
Some optimisation can be achieved through introspection, but ultimately, an llm can be wrong for the same reason that a person can be wrong, incorrect conclusions, bad data, insufficient data, or faulty logic/modeling. If there was a way to be always right, we wouldn’t need LLMs or second opinions.
Agentic workflows and introspection/cot catch a lot, and flights of fancy are often not supported or replicated with modifications to context, because the fanciful answer isn’t reinforced in the training data.
But we need to get rid of the unfortunate term for wrong conclusions,“hallucination” . When we say a person is hallucinating, it implies an altered state of mind. We don’t say that bob is hallucinating when he thinks that the sky is blue because it reflects the ocean, we just know he’s wrong because he doesn’t know about or forgot about Raleigh scattering.
Using the term “hallucination” distracts from accurate thought and misleads people to draw erroneous conclusions.
On undesired output, I would think it a great service to the field if we could come up with a better and earwormier word for “hallucinations” and somehow make it stick.
Right now we have half the literate world walking around thinking that LLMs are licking frogs, and it does nothing to help people understand how to think about model outputs or how to increase the utility of these fantastic culture / data mining tools in their own lives.
Encoding tokens from source text can be done a number of ways, byte pair encoding, dictionaries etc.
You can also just encode text into tokens (or directly into embeddings) with yet another model.
The problem arises that if you are doing variable length tokens, how many characters do you put into any particular token, and then because that token must represent the text if you use it for decoding, where do you store count of characters stored in any particular token.
The byte latent transformer model solves this by using the entropy for the next character. A small character model receives the history character by character and predicts the next one. If the entropy spikes from low to high they count that as a token boundary. Decoding the same characters from the latent one at a time produces the same sequence and deterministically spikes at the same point in the decoding indicating that it is at the end of the token without the length being required to be explicitly encoded.
(disclaimer: My layman's view of it anyway, I may be completely wrong)
Maybe it could eventually learn when it needs to have a low entropy token (to produce a more-likely-to-be-factual statement) and then we can finally have models that actually definitely know when to say “Sorry, I don’t seem to have a good answer for you.”
TLDR; RLHF results in "mode collapse" of LLMs, reducing their creativity and turning them into agents that already have made up their "mind" about what they're going to say next.
We can precisely determine the exact state of any digital system and track that state as it changes. In something as large as a LLM doing so is extremely complex, but complexity does not equal unknowable.
These systems are still just software, with pre-defined operations executing in order like any other piece of software. A CPU does not enter some mysterious woo "LLM black box" state that is somehow fundamentally different than running any other software, and it's these imprecise terms that lead to so much of the hype.
Zero temperature => fully deterministic
The neuron activation levels do not inherently form or represent a probability distribution. That's something we've slapped on after the fact
LLMs are basically "deterministic" when using greedy sampling except for either MoE related shenanigans (what historically prevented determinism in ChatGPT) or due to floating point related issues (GPU related). In practice, LLMs are in fact basically "deterministic" except for the sampling/temperature stuff that we add at the very end.
Yes, you can sample deterministically, but that's some combination of computationally intractable and only useful on a small subset of problems. The black box outputting a non-deterministic token is a close enough approximation for most people.
"The important thing to remember is that the output token of the LLM (black box) is not deterministic. Rather, it is a probability distribution over all the available tokens in the vocabulary."
He is saying that there is non-determinism in the output of the LLM (i.e. in these probability distributions), when in fact the randomness only comes from choosing to use a random number generator to sample from this output.
e.g.
In "black box" testing of a system you are just going to test based on the specifications of what the output/behavior should be for a given input. In contrast, in "white box" testing you leverage your knowledge of the internals of the box to test for things like edge cases that are apparent in the implementation, to test all code paths, etc.
However - even if, in principle, you could figure out any particular sequence of reasoning done by a model, it might in effect be "secured" and out of reach of current tools, in the same sense that encryption makes brute forcing a password search out of reach of current computers. 128 bits might have been secure 20 years ago, but take mere seconds now, but 8096 bits will take longer than the universe probably has, to brute force on current hardware.
There could also be, and very likely are, sequences of processing/ machine reasoning that don't make any sense relevant to the way humans think. You might have every relevant step decomposed in a particular generation of text, and it might not provide any insight into how or why the text was produced, with regard to everything else you know about the model.
A challenge for AI researchers is broadly generalizing the methodologies and theories such that they apply to models beyond those with the particular architectures and constraints being studied. If an experiment can work with a diffusion model as well as it does with a pure text model, and produces robust results for any model tested, the methodology works, and could likely be applied to human minds. Each of these steps takes us closer to understanding a grand unifying theory of intelligence.
There are probably some major breakthroughs in explainability and generative architectures that will radically alter how we test and study and perform research on models. Things like SAEs and golden gate claude might only be hyperspecific investigations of how models work with this particular type of architecture.
All of that to say, we might only ever get to a "pale gray box" level of understanding of some types of model, and never, in principle, to a perfectly understood intelligent system, especially if AI reaches the point of recursive self improvement.
That is, can you trace unequivocally the reason an LLM produced a certain token without, in effect, recreating the LLM and asking it the same question again?
Open models are almost always remotely hosted (or run locally) through a pure text-completion API. If you want chat, the client interacting with that text-completion API is expected to be the business layer, either literally (with that client in turn being a server exposing a chat-completion API) or in the sense of vertically integrating the chat-message-stream-structuring business-logic, logit-bias specification, early stream termination on state change, etc. into the completion-service abstraction-layer of the ultimate client application.
In either case, any slip-up in the business-layer configuration — which is common, as these models all often use different end-of-conversation-step sequences, and don't document them well — can and does result in seeing "under the covers" of these models.
This is also taken advantage of on purpose in some applications. In the aforementioned SillyTavern client, there is an "impersonate" command, which intentionally sets up the context to have the agent generate (or finish) the next human conversation step, rather than the next agent conversation step.
Huh, isn't it the other way around? Thanks to the academic (and open) research about LLMs, we have any open source community around LLMs in the first place.
The original ChatCPT was based on GPT-3.5, which did not use MoE.
But I wouldn't call the probabilistic interpretation "after the fact." The entire training procedure that generated the LM weights (the pre-training as well as the RLHF post-training) is formulated based on the understanding that the LM predicts p(x_t | x_1, ..., x_{t-1}). For example, pretraining maximizes the log probability of the training data, and RLHF typically maximizes an objective that combines "expected reward [under the LLM's output probability distribution]" with "KL divergence between the pretraining distribution and the RLHF'd distribution" (a probabilistic quantity).
In any case, while we know there's a transformer in the box, the operational behavior of a trained transformer is still somewhat opaque. We know the data flow of course, and how to calculate next state given current state, but what is going on semantically - the field of mechanistic interpretability - is still a work in progress.
That would be true only if all that we grant for based/true/fact came through reasoning in a complete logical and awoke state. But it did not, and if you dig a little or more you'd find a lot of actual dreaming revelation, divine and all sorts of subconscious revelation that governs lives and also science.
Even so the distribution of the second token output by the model would be stochastic (unless you condition on the first token). So in that sense there may also be a stochastic probability distribution.
You could still easily model the next token as a conditional probability distribution though if you wanted; the computation of entropy just might be a bit spendier.
Computing entropy generally does not.
The inference here is perhaps less pure than what you refer to but you're talking to human beings; there's no need for heavy pedantry.