The ultimate representation for an AI model is an ordinary computer program. Ideally, as a linear tape of instructions. Once we have that kind of a model at the frontier, I think the RSI monster becomes much more plausible.
https://www.rameznaam.com/p/471bbae4-1163-4048-944b-18f8b0bf...
No title editing please
The paper is about super-intelligence (not in the way people are claiming that term now, but in the "beyond human intelligence" sense), and the title here is about the "frontier", which by definition is the current SoTA.
Imagine you are evaluating what the machine should do when it is improving itself. It does a bunch of work and returns with "I supervaluated the liminal overdecomposition from the previous homological calibulation pass. It shows us that subtransitory mulutination will underspecify the tensor of stermullification. Where do you want to go from here?"
It will be like when you are reading a Wikipedia about a topic you don't understand. You follow the links, and you get more questions with more links. Your whole day is taken up following links, to the point where you forgot the original question.
Except this time, all the words come from the AI's work. You can't refer to an external authority who has already been there and can tell you what to do.
The AI needs you to tell it whether it is more intelligent than it was before, but you don't know, because you can't follow its reasoning any more. It's like an ordinary person trying to hire a math professor, there's just no way to do it.
But whereas a human math prof can evaluate another one, a machine intelligence can't evaluate another one, by construction. Because it's still usefulness to humans that is the evaluation criterion.
It's worth pointing out that so many people already see and use AI the same way. In such a situation some would ask AI to provide options, and they would choose and experiment with those options. Of course, contexts/stakes can be vastly different.
Given that there are so many phenomena in nature that we can't explain or fully understand which doesn't prevent them from existing or being useful, there could be a future where humans accept the same about the things AI comes up with as long as this leads to desired outcomes. We still might have names for them, but our brain thinking/knowledge capacity wouldn't allow us to fully comprehend them. We'd need frameworks/systems in place to turn these AI features/decisions on/off, although it's hard to imagine how this wouldn't increase the likelihood of something going out of control.
For it to have new vocabulary we dont understand, it needs to have novel ideas that need words coined for them, and a way to persist those ideas and words into the future. I don’t think that exists.
To me this hypothetical make it clear this won’t happen, not unless there are fundamental changes to what llms are. It doesn’t suggest it will happen. To me, anyway.
Theoretically you could update the weights at inference time too though so the model evolved as it's used. Surely some people are trying this already.
But given our current relationship even if it did I can't foresee a point where it couldn't walk us through the necessary steps or supply the pros/cons for whatever problem is being addressed.
2 issues - trying to understand how it came to its conclusion because I feel true AI has got to be non-human intelligence. Or a something catastrophic happens and we as a species are back in the stone age. Imagine today's AI trying to converse with a cave man (yes, one without modern languages even).
We can think of questions we can't answer. It can answer them.
[1] https://www.noahpinion.blog/p/wheres-the-intelligence-explos...
... provided you keep eyes closed when examining the output.
To those gulled by the AI coding hype, I suggest try a trivial code task with e.g. immediate feedback in a domain you can swiftly verify e.g. HTML or SVG. Your red pill.
Example: https://chatgpt.com/share/6ac0d871-6314-83eb-990a-41baed9642...
It’s like 10 monkeys trying to keep a human being a prisoner.
It’s just not going to work is it.
The entire meaning of being more intelligent is being able to outwit and out-think people less intelligent.
So do you also believe that Kim Jong Un is the most intelligent man in North Korea?
Are you saying it will be OK because we’re not as clever but we are more brutal than the super intelligence so we can still control it?
We think we can introduce something cleverer than us and still control it. That makes no sense.
With that argument, the basic calculator would have been impossible.
It's pretty clear that machines can be better than the best humans at memorizing things, computing relationships between concepts quickly, etc. So while you can argue that LLM are not the path forward, I have no problem imagining that using all of humanity's knowledge with a machine could beat the best human at a field.
Deep NNs and LLMs should not work based on our theoretical understanding. The fact that they do should give us a pause instead of us flatly denying their unexpected performance.
Tasks like using a CNN to detect digits has been well understood since the 2000s. The explosion in the capability of LLMs is very surprising, sure, but where is the concrete proof that such systems "should not work"?
token != step.
Just you try executing a complex command one word at a time.
They're doing RL on open problems these days, not just next token prediction.
...but many orders of magnitude faster, and in a way that scales horizontally really well, which is quite useful even if the quality isn't quite what a the absolute best humans can do.
One way AIs really really really excel is pulling together a lot of different data sources and reasoning over that data. In the past sure we could collect data and create huge datasets, but the analysis of that data - extracting themes, finding commonality or issues etc - either required extensive human research and analysis at best, or at worst crude regexes or keyword matching.
Now an AI can pour over that data and make its own inferences and decisions and findings that we've simply not been able to do before at this kind of speed or scale just because of time and resources.
And the AI, having done that, can propose new things for e.g. training, i.e. new things that no human has ever done before that the AI is simply repeating. For example it can propose a task that it knows from it's research is hard for it to solve currently, and then we just throw compute and randomness at it to find the "best" solution from many many attempts, then repeat until we hill-climb up to a perfect 1.0 score (... although of course we have to try and avoid cheating/attempts to short-ciruit the eval)
So this could be coding tasks, UI control tasks, protein folding, maths, chemistry etc etc. Anything that is easily and objectively programmatically scored. You can run this in a loop many times, each time you go around the loop the model gets smarter, learns more things from it's research, new areas of loss it can optimise etc etc.
It's harder where there is not a way to objectively score the outcomes (e.g. art, creative writing). Often this uses a fuzzy "judge" model that is trained specifically to give the work a score based on it's appraisal. This works but you can see how we might end up with feedback loops, so often it is paired with humans who provide feedback to provide supervised fine tuning datasets.
Tl:Dr - It's not just "repeating what it's seen". AI is finding new ideas and creating new things millions of times a day, and that is just software engineers asking it to write code or fix bugs, let alone people using it for actual research or whatever.
You think wrong ( https://dl.acm.org/doi/10.1145/3442188.3445922 ) ... except about irritating. Yes, its irritating to people insisting next-token predictors are intelligent.
We therefore control all the ants, all the bees, all the tiny living dumber things that are too small for us to see ...
That's not actually true though, is it?
Counterpoint> How did that work out for intelligent people in the past? eg: During the Pol Pot regime (and other examples).
To me, it is not at all obvious that the "level" of the training set is an upper limit to the capabilities of an LLM.
Sure, the LLM hasn't been exposed to material more advanced than the most capable human domain experts have produced. However, it has seen and learned from a vast amount of information that these domain experts are completely unaware of. Why shouldn't the LLM be able to use that information to produce output that's beyond the capability of a domain expert?