Compression is prediction(ngrok.com) |
Compression is prediction(ngrok.com) |
> Compression, Predictive modeling, or Complexity?
Perhaps a bad example: https://news.ycombinator.com/item?id=38400380 :
> "78% MNIST accuracy using GZIP in under 10 lines of code" (2023) https://news.ycombinator.com/item?id=37583593
> Why unify information theory and machine learning? Because they are two sides of the same coin. In the 1960s, a single field, cybernetics, was populated by information theorists, computer scientists, and neuroscientists, all studying common problems. Information theory and machine learning still belong together. Brains are the ultimate compression and communication systems. And the state-of-the-art algorithms for both data compression and error-correcting codes use the same tools as machine learning.
Book (creative commons): https://www.inference.org.uk/mackay/itila/book.html
Lectures: https://m.youtube.com/playlist?list=PLruBu5BI5n4aFpG32iMbdWo...
This post references specific parts/calculations, but you'd never know it was not news if you didn't know better.
I don't think it's the case here, but worth noting too that LLM-written blog posts adopt this tone seemingly by default.
Never the least bit of surprise, wonder, doubt, or frustration to get in the way of the steady staccato beat of metaphors, conclusions... and three-item lists.
Any rigorous CS program should cover this in depth.
[0] Compression is Intelligence Part 1 - https://youtu.be/l6DKRf-fAAM?si=yyLWq8x4sSRkWd98
So they're both sourcing a bit broader zeitgeist.
Compression is functionally equivalent to prediction when the data distribution is exactly representative of all future problems. The story changes drastically if you want generalization -- because the test distribution could be arbitrarily different, even if it had the same support! Eg: you observe a rare edge case in your training data and (lossy) compression could simply ignore it. But if you wanted generalization in that particular part of the space -- either because an adversary was testing you, or for design freedom where you choose to build in that specific corner -- then you don't just want data compression, but good prediction performance on a test distribution which peaks in that corner.
Assuming that the training data distribution is exactly the distribution you will ever care for is implicitly doing a lot of the heavy lifting in the claim that compression = prediction, and I'm peeved at how much this statement is unthinkingly repeated like a manifesto.
There is nothing natural about the training data distribution, especially if the data generation process is exploratory while the downstream usage will be exploitative.
Say you have a computer file composed of two parts, the first represents the setup of an experiment, and the second is the data produced by the experiment.
If you have a good theory relating to this type of experiment, then you can predict much of the second part of the file. So you only need to store the first part and possibly some corrections to the least significant bits of some of the parts of the second part of the file.
Thus with good prediction, you can compress this type of file.
But if you had a predictor so smart that it could crack the encryption key, it could start predicting the rest of the encrypted stream, and therefore compress it.
Because most of the training data has good grammar, the LLM is strongly trained on grammar; it will rarely predict ungrammatical gibberish. Even if there are grammar mistakes in the data, they are not systematic and so don't reinforce each other.
However, going from compression to prediction is a large jump that is unsubstantiated by this article and based on the claim that probabilistic recall is also prediction.
Two perfect counterpoints to this are markets and weather patterns. One cannot predict future events based on past performance or behavior. Change is the only thing that's constant, and chaos/entropy is everywhere we look.
For simple problems like programming, sure predictive recall works amazingly well, but let's not pretend LLMs are actually predicting something. This is exactly why LLMs suck at doing anything novel; they lack imagination and creativity.
https://www.newyorker.com/tech/annals-of-technology/chatgpt-...
Deflate (as used in gzip) uses a Huffman coder. LZMA (as used by xz) uses a predictive range coder. Zstandard can use either Huffman or FSE. Some high-speed compressors like LZ4 skip the entropy coding stage entirely at the expense of compression ratio.
Bzip2 is an interesting aversion of this pattern - it uses the Burrows-Wheeler transform as a first pass instead of LZ. Unfortunately, this is one of the major reasons why it's so slow.
I found this interesting and wonder whether LLMs have a higher density ceiling, since training and inference don't rely on a fixed lattice and can instead learn their own representational geometry.
JPEG's zig-zag is a primitive for quantization, throwing data away based on rough approximation of human perception and biology. That isn't compression itself. However, the rounded and zeroed-out data is then compressed using a combination of RLE and Huffman, set up to predict the data will have lots of zeroes and few other distinct values (which the earlier step forces to be true). Or if you think about the system as a whole, you could say that JPEG predicts images will be blocky low-frequency patterns of DCT.
There is the Kolmogorov Complexity [1], Normalized Information Distance [2] and Normalized compression distance [3] that correlates those.
Finally, there's the Pre-Big Bang Informational Compression and the Delayed Release of Antimatter [4]
All big {rabbit/black} holes to lose some time, if you have any.
[0] https://en.wikipedia.org/wiki/Prediction_by_partial_matching
[1] https://en.wikipedia.org/wiki/Kolmogorov_complexity
[2] https://homepages.cwi.nl/~paulv/papers/chapter08.pdf
[3] https://en.wikipedia.org/wiki/Normalized_compression_distanc...
Another thought that came from the same post is that, insofar as we see LLMs as human-style intelligence, they're more like stream of consciousness devices. Essentially incessant talking and buying enough time until you get to a usable answer. I think I associate some subset of intelligence with what you don't say, which is impossible with the SOC-style outputs, so this is something I think about a fair bit.
What could maybe differentiate current gen models from next gen is the ability to call tools modeled within the layers themselves, not externally. I think as far as I understand it, model trainers expect the model to do this itself in a way we don't understand or control, like a version of the bitter lesson. But I posit we can model many determinate tools as NNs themselves and figure out how to get the internal states of the LLM to make use of them during inference, e.g. calculators, indexes, citations. Just an enthusiast though, so grain of salt and all.
Specifically I was thinking about a way to inject knowledge into LLMs training by using statistical properties of text in such a way that you don't have to train the LLM to achieve some level of predictions. There are actually some papers that inject n-grams statistics as a part of the neural network weights.
The article itself has decision trees for the compression explanation, which is also a lookup index.
In each case you try to recognise (re)usable structure.
Self-indexing succinct data-structures are a good example of the third side of the coin.
So it's a trinity: compression, prediction, indexing
Why only LLMs? All statistical models are compressor. You can say "model" and "compressor" are synonyms.
Article does not mention "embeddings" at all, even though it's commonly viewed as a compression method. Also "encoder" part on "auto-encoders".
Hutter Prize being where you are paid if you can compress wikipedia small enough. LLMs do very well at that, if, big if, you ignore the cost of initial weights.
A cool Claude Shannon story:
Shannon wanted to measure how much information is actually contained in ordinary
English text. His 1948 theory said such a number must exist, but he had no way to
calculate it, because the patterns in English reach across dozens of letters and no
equation or frequency table captures all of them at once.
So instead of calculating it, he ran an experiment on a person.
He took a passage from a novel that the subject had not read, and covered it with a
card so only the text already guessed was visible. He asked the subject to name
the first letter. If the guess was wrong, he asked again, and kept asking until the
subject named the correct letter. He wrote down how many guesses it had taken,
revealed the letter, and moved the card one position to the right. Then he repeated
the process for the next letter, and the next, through the whole passage.
What this produced was not a sequence of letters but a sequence of numbers — one
number per letter, recording how many guesses that letter required. Most of the
numbers were 1, because someone fluent in English, seeing the preceding text,
usually names the next letter correctly on the first attempt.
Shannon then argued that this sequence of numbers contains exactly as much
information as the original passage.
https://corecursive.com/the-hutter-prize/
LLM embeddings are compressed training data.
To decompress that is to make a prediction (in this case to convert the embedding into readable text)
I'm not a LessWrong^TM rationalist guy, but one really good thought experiment I always keep in the back of my mind from them is Solomonoff induction. AIT people take it as a framework to work with - it's pretty cool, I agree. But I (and some other people, such as certain AI execs at Amazon - according to my interpretation of their public interviews) think it just highlights the trap - given an arbitrarily powerful oracle, you can get compression down pat. Like, if you assume the source is generatable with a turing machine, and you write a function to brute force over all turing machines, then whoa, your compression works. You will necessarily find the optimal compression at some point because your search function is literally searching over all possible turing machines that could've generated the input sequence, anyways (because the input sequence was generated by a turing machine)
These are the kinds of results you can get if you don't have any actual constraints on what the compressor can do.
(Of course, again - this is not the point of solomonoff induction - it's to use this as a base truth, to then layer parsimony on top of that. There are infinite number of turing machines that could match your prefix, parsimony filters, throw some bayesian inference on top of that, and you get Solomonoff induction. They constrain it afterwards. But I think to that intuition as a base whenever people claim new results.).
But I see in casual conversation, people constantly making claims like, "LLM's are so good because they compress a model of the world". What is that model then? Scott Aaronson has made points like this before - your "model" could just be a massive lookup table, so you can't just claim "compression" and win - the compressor must be reasonably small, too.
I don't object to the notion that LLM's have some notion of world models more sophisticated than memorization. That's proven by actual interventional experiments, such as the ones that actual interperability researchers do. But mere compression is vacuously powerful. "Vacuous" not in the sense that "oh, you might be suboptimal and be a little more complex", vacuous as in "the philosophical point you were trying to make is vacuous because you make a vacuously powerful statement".
(I'm not a total fan of intervention either, as an end-all gospel as some people use, but it's far, far better than not having it).
> Annie Sexton is a Developer Educator at ngrok with a passion for nerd-sniping developers.
For instance, consider the distribution of strings drawn from the language '0+'. Now consider the same for the language '[01]+'. A compressor looking at only the strings of the first language within those of the second can do a much better job if it does not have to account for future data.
This also relates distantly to the idea of overfitting in machine learning.
At some level, nothing is representative of all future "problems".
I think as far as corner cases is concerned, people can optimize for what they think matters. Larger models, higher sensitivities, better filtering.
Beyond this, I don't know what you're peeved by. There's no manifesto, there's no claiming that any particular method to prediction is perfect yet.
Predictors don't have to give you some corpus-average probability. They can give probabilities adjusted per symbol. The probability could even be exactly 1 for some symbol when the predictor is absolutely certain (and then it costs 0 bits to encode).
If you had a magic oracle predictor that already knew all the data you wanted to send, you'd have perfect compression.
Also sparked the thought that the assumption only holds if the future looks like the present.
This is not intuitive to me. It seems like a "new idea" is something that (almost by definition) isn't in the training set. Can you elaborate a bit?
Edit: but perhaps a good model could arise from training, which would be a good idea in the sense that parsimonious ideas are good scientific ideas.
Of course, one thing you get out of this is a great curve-fit for your existing data, which you can interpolate to find the position of any observed planet at any desired time.
But could this function also succeed in predicting the orbital motion of objects that aren't in the dataset? If I spot a new comet, and put it into the compressed function, would I get an accurate prediction of its orbital motion, even though that object wasn't in the training data?
The answer is "it depends, but probably yes". Newton's laws of orbital motion turn out to be simple compared to the size of the training data. So if the black-box compression has done a good job, it might output that function, or a close approximation of it. With a sufficient quantity of sufficiently accurate data, it might even improve on it; random errors can't be compressed, but where the deviations between observations and Newton's law turn out not to be random but rather the influence of an unobserved gravity source, or general relativity, the black-box algorithm will likely capture that as well.
A lot of people seem to think of the training process as curve-fitting data (the "stochastic parrot" model), but I think of it more as "solving an inverse problem to approximate the unknown source that generated the training data". Machine learning has proven to be quite good at solving inverse problems, and this is just a very abstract one of them.
(A forward-problem is something like solving for the electric fields from a set of charged particles; an inverse problem, https://en.wikipedia.org/wiki/Inverse_problem, is one where you have data on the electric fields at various points and want to reconstruct the arrangement of charged particles that produced it. Or more generally, you have sampled data on the output of an unknown process, and want to reconstruct the process that produced the data).
The inverse-problem-solving happens at the ML training step. The language model itself, that comes out of that, is solving the forward-problem: it has a generative-process baked in and now it's generating new data from it. But if the training process has done a good job of compression, it will certainly be able to generate valid new ideas that aren't in the training set, because the inverse model has solved for the underlying features of the real process that generated the training data, and those features can generate additional valid outputs that it wasn't trained on.
I don't think Apple invented the ipod anymore than I invented it; LLMs likely would have also come to the same conclusion about an ipod like device.
Original ideas either dont exist or have a functionally irrelevent definition in comparison with inputing tokens to LLMs to get novel ideas out.
The exception being pure mathematics since it exists solely in the realm of ideas. I'm willing to call that knowledge, but it's still a distinction, the old analytic/synthetic dichotomy of Kant.
I think this is relevant to the discourse on LLMs/programming because for months, people said “they’re just regurgitating their training set,” but now I think people are seeing (I am seeing) that they do learn more abstract models of the world than that. I don’t really know how, but it’s why they can generalize from other codebases and tools and so on.
Stock market is even more pathological than weather here. Most statistical tools have baked-in assumptions about some degree of independence between factors under analysis. In weather, things are hard, because everything affects everything but in some random sense. With stock market, it's worse, because some things affect others in an adversarial way. A pattern discovered is not merely drowned in noise, but rather actively canceled out as other people discover and act on it. The way to describe this in formal terms is, I believe, that the stock market is "anti-inductive".
Fortunately, most things we deal with life are much friendlier to predictions than weather or stock prices.
> I think people are seeing
Better late than never, given that this has been empirically demonstrated two years ago by Anthropic. I actually don't understand why people don't make a bigger deal out of it - maybe it's because the AI hype/hate fight is sucking all attention from the room. But the point is: we not only have a thing that can learn concepts we recognize and build abstract world model, but also we have a degree of visibility and control of it we never had before. We can't see, much less suppress or stimulate individual concepts or concept groups in a human brain directly, but we can with LLMs.
> A prediction is a statement about what you think will happen in the future, often based on experience or knowledge. It can also be referred to as a forecast or an informed guess
Based on my reading of this definition, compression may inform prediction but it is not itself prediction. The examples cited in the blog post are examples of probabilistic recall based on past events or instances. More context means a higher chance that the recall is more likely to be aligned.
But it's hard for me to accept the leap to compression == prediction because in my mind a prediction is an informed guess about something that hasn't yet come to pass. But thinking more about it, time is a human concept and so who's to say the temporal reference means anything at all here. Maybe probabilistic recall is the same as predictive forecasting if time is an invented concept and essentially means nothing?
Is everything fundamentally deterministic if you know everything in the universe or does free will exist?
IDK to be honest, I'm just more frequently surprised by new things that happen every day than I am at things that stay the same, even if mostly things stay the same. Maybe I just don't notice them and nothing actually ever happens.
Side note: the inevitable consequence of this line of reasoning will eventually become that LLMs given enough power are in fact intelligent and sentient, and I'm worried about how that affects humanity as a whole. Are we about to subjugate the most intelligent thing humanity has ever created, or is it about to subjugate us?
The rabbit hole gets deep quick when making the leap between a fancy recall mechanism and novel prediction, but I agree they're not that different in the end. I just believe it's important to be nuanced or else we'll miss when AGI actually happens (maybe it's already here).
Given a context (for LLMs, this would include the entire pretraining dataset, plus the prompt), you compress `context + next_token` for every possible next token. The tokens that co-compress best with the existing context are the 'least surprising' continuations. Choose one of them and iterate.
You can easily generate text with gzip this way. It won't be very good text, because gzip compression is not as sophisticated as a transformer + SGD, but the principle is the same.
You can, actually! Any compressor can be losslessly converted into a generator, and vice versa.
Traditional compressors like gzip are of course very simple and can only replicate rough patterns from the input. But they are technically doing the same thing.
Makes me wonder idly, - is this conceptually akin in some sense to a "universal grammar," and if so - with a broad enough training set, is there a latent durable universal grammar that might be similarly recovered and injected to the benefit of all training, - does that grammar go beyond morphological/syntactical/grammatical features, into e.g. semantics and pragmatics
But actually these techniques are used but they are hidden as speculative decoding with increasing complexity of approximations. For example you can have a part of the network that predicts the next word based on the markov chain, the next approximation is more complex etc.
This paper proposes something similar where you can inject memory without training https://arxiv.org/abs/2605.16893
If you want to best predict what state comes next from a space of possibilities, you have to figure out how probable each next state is and pick the most probable one.
If you want to compress something, you have to figure out how probable each next next state is and assign the smallest code to the most probable state.
These two processes are essentially the same, and the resulting structure of a system which regulates either process will be similar, approaching the same structure at high confidence.
>> a Developer Educator at ngrok with a passion for nerd-sniping developers.
Maybe more the latter than former...
I just wanted to confirm your underlying point here: training a model isn’t about finding a function that fits the observed data (even though that’s the outcome) but instead finding an approximation of the unknown source that generated the source data in the first place.
In the case of LLMs an approximation not of a human but of the sum of humanity that produced the training copora?
When you shorten a description in a lossy way, you are deciding a priori that some differences in the object don't matter, and it's not because you understand the object, but because it serves your goal of shortening the description.
You can't just assume smaller is better but it often is. And very often it's more information-dense.