Self-Adapting Language Models(arxiv.org) |
Self-Adapting Language Models(arxiv.org) |
Two things that stand out:
- The knowledge incorporation results (47% vs 46.3% with GPT-4.1 data, both much higher than the small-model baseline) show the model does discover better training formats, not just more data. Though the catastrophic forgetting problem remains unsolved, and it's not completely clear whether data diversity is improved.
- The computational overhead is brutal - 30-45 seconds per reward evaluation makes this impractical for most use cases. But for high-value document processing where you really need optimal retention, it could be worth it.
The restriction to tasks with explicit evaluation metrics is the main limitation. You need ground truth Q&A pairs or test cases to compute rewards. Still, for domains like technical documentation or educational content where you can generate evaluations, this could significantly improve how we process new information.
Feels like an important step toward models that can adapt their own learning strategies, even if we're not quite at the "continuously self-improving agent" stage yet.
"NEAT/HyperNEAT" (Neuroevolution of Augmented Topologies) [0]
I'm no ML practictioner, but as I understood it, the primary difference between NEAT and what is described in this paper is that while NEAT evolves the topology of the network, this paper seems to evolve the weights.
Seems like two approaches trying to solve the same problem -- one evolving networking structure, and the other the weights.
Those 2 friends are quite possibly the most intelligent people I've ever met, and they were very convinced that RL and evolutionary algorithms were the path forward in ML.
[0] https://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_t...
SethBling's MarI/O - Machine Learning for Video Games
Finding a good scoring algorithm is hard as it is so easy for a GA to cheat...
Source: experience
"when assessed by Claude 3.5 Sonnet’s production-grade RM, our unsupervised assistant policy wins 60% of head-to-head comparisons against the policy trained with the human-supervised RM." So now the models can even post-train the new models better than a human can
Unsupervised Elicitation of Language Models - https://news.ycombinator.com/item?id=44276041
I’m sure this is something the big labs are trying but from the outside as a user of LLMs it feels like people don’t talk about this very much and instead the focus right now is on better training (eg reinforcement learning) with the assumption that anything else not learned during training will be stuffed into the context somehow as needed. But from a naive perspective the lack of learning from experience after training seems like the biggest thing standing between us and AGI.
I'd love to see a full circle of hypernetworks, with both models continuously updated through generated LoRAs, the hypernetwork updated to accommodate the new model state. You'd need a meta-hypernetwork to apply LoRAs to the hypernetwork, and then you could effectively have continuous learning.
The learning and inference process are entirely separate, which is very confusing to people familiar with traditional notions of human intelligence. For humans, learning things and applying that knowledge in the real world is one integrated feedback process. Not so with LLMs, we train them, deploy them, and discard them for a new model that has "learned" slightly more. For an LLM, inference is the end of learning.
Probably the biggest misconception out there about AI. If you think LLMs are learning, it's easy to fantasize that AGI is right around the corner.
In any case, this is a far cry from what I was discussing. At best, this shows an ability for LLMs to "learn" within the context window, which should already be somewhat obvious (that's what the attention mechanism does). There is no global knowledge base or weight updates. Not until the content gets published, rescraped, and trained into the next version. This does demonstrate a learning feedback loop, albeit one that takes months or years, driven by external forces - the company that trains it. But it's way too slow to be considered intelligent, and it can't learn on its own without help.
A system that truly learned, ie incorporated empirical data from its environment into its model of the world, would need to do this in millisecond time frames. Single celled organisms can do this. Where you at AGI?
"Forgetting correctly" is something most human brains are exceptionally good at, too. I wonder how that works...
2028 is pretty much tomorrow… fascinating insight
Many people here are right, compute, collapse, forgetting whatever.
The only "real" way to do this would be: 1. Train a model 2. New data 3. Retrain the model in full + new data 4. Repeat 5. You still have no garuntee on the "time" aspect though.
But CL as a field basically has zero answers on how to do this in a true sense. It's crazy hard because the "solutions" are hypocritical in many ways.
We need to expand the model's representation space while keeping the previous representation space nearly the same?
Basically, you need to modify it without changing it.
Most annoying is that even the smallest of natural brains do this easily. I have a long winded theory but basically it boils down to AI likely needs to "sleep" or rest somehow.
LoRA paper: https://arxiv.org/abs/2106.09685
1. Preventing collapse -> model gets "full" https://arxiv.org/pdf/1612.00796
2. Forgetting causes better generalization https://arxiv.org/abs/2307.01163
3. Unknow paper that connects this - allow a "forgetting" model that improves generalization over time. - I tried for a long time to make this but it's a bit difficult
Fun implication is that if true this implies AGI will need "breaks" and likely need to consume non task content of high variety much like a person does.
I completely agree that figuring out a safe way to continually train feels like the biggest blocker to AGI
Hypothetically (and perhaps more plausibly), a continually learning model that adapts to the context of a particular org / company / codebase / etc., could even be desirable.
There are tons of benchmarks around this you can easily run with 1 gpu.
It's compute only in the sense that the only way to do it is retrain a model from scratch at every step.
If you solve CL with a CNN you just created AGI.
The other thing is the brain down values and prunes paths we don't use and strengthens one's we do. This is why something you've not done it a while might need a refresher for you to do right again.
This is often associated with learning tools like anki and stuff, but the real world is all about encountering things at certain frequencies (day night cycles, seasons, places you visit, people you see.... everything, really)
I'm wondering if there maybe some sort of inverse to SR, maybe?
They don't just "forget" that information can come back at a later time if you continue to train.
So basically any time a model is trained you need to check it's entire memory not just a small part.
No one in the research and science communities ever said anything contrary to this and if they did they wouldn't last long (although i imagine many of them would find issue with your stochastic parrot reference).
The apple paper has a stronger title than its actual premise. Basically they found that "thinking" definitely works but falls apart for problems of a certain difficulty and simply scaling "thinking" up doesn't help (for these harder problems)
It never said "thinking" doesnt work. People are just combining the title with their existing prejudices to draw the conclusion the _want_ to see.
Online learning is more akin to RL in that it's a structured and boxed enviroment. Step outside of that box or the box changes too much and you collapse.
CL is much more similar to meta learning. The concepts are more about learning NEW content while keeping previous the same.
CL is a completely open problem with all model types. EWC is amoung the better attempts (and a favorite of mine) at solving it with big limitations.
They try to keep a single continuous "state" that always updates.
It's more about going "farther" than something more akin to "go forever" that CL promises.
Scaling laws are true in that infinite scale would 100% lead to AGI. But at the same time the problem with it is that you can't infinitely scale the computation per task.
RL solves this problem in general but it has a deep assumption of knowing the future. Step too far out of the box and it collapses.
The smallest natural brains handle unknown future states with a fixed computation budget per timestep which is truly incredible.
does a model need to retain its generality?
Only if you want it to remain smart.
But it doesn't solve the time aspect.
You need to randomize data in order to train to best quality. In doing that the model has no idea t0 was before t1000. If you don't you get model collapse or heavy bias.
Some attempts at it but nothing crazy effective.
But the clone couldn't run without sleeping? So that's more of a teammate than a clone.
1 works while the other sleeps and then swap.
If this method ever worked our current alignment methods get chucked out the window those would be two completely different AI.
CL has been an open problem from the very beginnings of AI research with basically no solution. Its pervasiveness indicates a very deep misunderstanding on our knowledge of reasoning.
I just like talking about it. I will say that learning outside distribution content while keeping previous knowledge in a "useful" state is a capability that would absolutely supercharge ever AI method we currently have.
It's atleast an honest atempt at a research direction other than "scale infinitely for everything" that we currently do.
Just think about how natural brains do something incredible.
1. They have fixed computation budgets per time step. 2. They continously learn entirely new tasks while still maintaining previous in a useful state.
That's a capability I would very much like in my AI.
Scaling laws are correct but they are also the reason we are nowhere near replacing humans.
Take a simple job maybe admin work. Every timestep depends on the previous timestep. While not a complex job and an AI could do it for awhile but over time the compuation required to "look back" its memory and connect it for the next step grows near exponentially.
RAG is another perfect example of this problem.
I do deeply belive AGI will be solved by a kid with a whiteboard not a supercluster. CL is my best guess at what that means.
Maybe it's a super RL or energy type method but I've never seen it.