Is would be understandable for it to be wishful thinking and dissociation with reality facing the fact that our jobs and what many of us love doing is being automated and taken away, and our craft being made economically worthless.
But no one knows where the limit is. It was only one year ago that Claude code was starting to become useful and now it handles large code bases with a dexterity that is unseen in humans, where within each session it basically starts from a clean slate.
So instead of denying that this is actually happening right now, it would be much more productive to actually take sama and others at their word and push for global regulation, to force them to do as they say and slow down with this technology and proceed more judiciously.
I think AGI will be eventually achieved, but it will not come from an LLM, and it will not be anytime soon.
I'll worry when a more promising architecture comes online.
And it didn't even start with code, but just from being incredibly accurate and precise at predicting the next token. From that emerged being incredibly good at handling code, and being incredibly good at finding counterexamples for major open mathematical problems.
The thing that these agentic systems are getting incredibly good at is steering outcomes, that is to make the world behave in a way that fits their objective. We had that with the Go AI on a game board, now with code on a computer and with mathematics in an abstract world determined by axioms.
And then people just say "Ah, because sama said we should Pause this means that this is it, the bubble is popping". I wish there was a real, strong argument for why ASI will not be achieved by LLMs, and why it will never be able to steer outcomes in the real world. But so far the disciplines just keep falling as LLMs are scaled further, and there seems to be no limit in sight.
https://xcancel.com/cocktailpeanut/status/209733229184439951...
This is just their way of saying they're officially switching from providing AI to using the AI to compete with their customers, a la Navier-Stokes drama.
Perhaps it's getting more difficult to move the 'goal posts'?
- Train LLMs on texts -> OMG AI can write!
- Train LLMs on code -> OMG AI can code!
- Train LLMs on images -> OMG AI can generate images!
- Train LLMs on video -> OMG AI can generate videos!
- Train LLMs on math -> give free access to 100,000 researchers/scientists/mathematicians -> invest resources into Lean -> OMG AI can solve math problems!
I say 'goal posts' because that's a common argument, but the perspective that seems to not be considered is that the LLMs are doing the same thing over and over again in different domains while be fine-tuned by labeling and human review.
The emergent behavior argument is a separate consideration, but I can imagine an AI being given a list of words (a harness) that guides how it operates--not the other way around--and the difficult problem is narrowing down and filtering all the associations with the words/harness to properly align it.
And perhaps the recursive looping being discussed creates incredibly difficult (impossible) to predict behavior? If the AI could modify it's own harness, is that AGI? If we fully understood what happens in high dimensional space, and could predict the output of any combination of input tokens, then is it not AGI?
Or perhaps I just tell myself this to stop worrying about a future involving ASI (:
I guess now we do.
1. What are we actually talking about, slowing development or slowing public release? I can't imagine the the US is keen to slow actual development when we have adversarial state-backed labs continuing development. Seems to me that this is a global pact or it's nothing. Stopping a single lab, even if it's the current leader, won't put the genie back in the bottle.
2. Unless they stop actual advancement, isn't the risk essentially identical? At this point, it's an arms race between the lab and its own model. I don't think it far fetched to think that it might successfully exfiltrate its weights, for example.
3. There are going to be factions who don't want to stop. Should we worry about them defecting to other labs or other countries?
The problem is with that much debt buried in the report, no one is buying peoples sandbagging nonsense anymore. Even the boring bond market is showing odd behavior much to the chagrin of congress. =3
If progress is coming easy and fast and you're worried about the danger your product poses, you should also be worried about the danger your competitors' products pose. So, you reach out to governments and ask for rules to constrain and slow things down for everyone, so that your company isn't put at a disadvantage for being cautious.
When all labs are approaching the same marginal improvment which is a clear sign that more breakthroughs needed to achieve the god they hyped.
"BUY BUY BUY BUY BUY BUY BUY"
With other countries and open source, not to mention all the other companies, Open AI choosing to pause development wont change the course. Only OpenAI will lose.
Unless the secret is that no amount of compute or data can progress things any further because we have already reached the limit for what predicting the next token can do. Most of the advancements lately have been on the agentic side of things.
The massive amount of health, science and math problems being solved by AI has largely not materialized yet. Is this a sign the model capability has been over sold?
There are economic consequences to this as well because the data center build out has been a kind of circular financing. This has caused all kinds of macro economic impacts from increased prices to even the Treasury market itself having to compete with debt issued for the build out. No one knows what will happen once the data center build out stops, or if the yen carry trade continues unwinding, or if Treasury yields have to keep rising to compete with debt issued by the "Mag 7".
Once they stop spending in the current circular arrangement, then the profits that exist on paper disappear, and the bubble could pop.
So really for every reason, OpenAI cant really slow development. Unless there is a kind of Moores law for what predicting the next token can achieve.
Either way it's a win-win for whoever sells that message because you make the makers and the warners the center of attention. And when they say "we are considering slowing down" they mean "if we get enough money to slow down we'll do it". And of course they never want you to think AI is a technology employed by humans who need be changed, not the technology. So as long as you engage this massive straw man to focus on the wrong thing you do nothing to hold the leaders accountable for their harmful and selfish actions and decisions for society.
They won't. There is no slowing down this development anytime soon and no one will slow down for someone else to surpass them.
There's no way to know what is "safe enough", so you might as well go as fast as technology will allow, and be on the watch out for negative things you can predict specifically and then adapt so as to minimize net negatives.
which is to say - unlikely for it to materially slow, only a lot of chirping about slowing down to the public with a couple outlier examples of 'slowing down' they can show publicly
- 1386 employees of frontier labs called for it
- lawmakers are calling for it
- employees are quitting in protest
All it needs is coordination to regulate the pause
"Development of superhuman machine intelligence (SMI) is probably the greatest threat to the continued existence of humanity. There are other threats that I think are more certain to happen (for example, an engineered virus with a long incubation period and a high mortality rate) but are unlikely to destroy every human in the universe in the way that SMI could." - Sam Altman
"If Anyone Builds It, Everyone Dies": https://en.wikipedia.org/wiki/If_Anyone_Builds_It,_Everyone_...
So? The upper end of the S-curve is also a line that still rises.
For a superintelligent AI trained with methods anywhere close to what we are doing now, killing all humans is the default outcome. I actually think researchers are downplaying the risks to make their arguments more normie-friendly.
Serious question - I want serious responses pls ty.
The US hasn't fully woken up to the fact yet that the race to superintelligence is a death race. China (population, members of the government, academics, lab employees) could also just not have fully grasped it yet. I'm personally confident that it will be possible to explain it to them.
Perhaps CFCs (Montreal Treaty) is a better example of international cooperation to solve a potential extinction-level event, though it was never a "race" in the same sense.
I am curious what it would look like if the shoe was on the other foot, though. If China had the lead, and Qwen 4 was an Astra class model and the best US model was only as good as Kimi K3, would US commentators call it AGI?
I expect they will continue optimizing and improving for the current use cases/benchmarks. But the core capabilities will stagnant, that's why they will be forced to slow down until another major breakthrough happens.
I personally don't think LLMs have any understanding of the world, nor any imagination or even agency. I think it got very good as recognizing patterns and shapes in human thinking, language and knowledge, but that's pretty much it.
I would love to see someone try to escape with a copy of openai's raw unaligned base model
the things are kept under air-gapped national weapons grade security measures not because it's literally going to escape and threaten the world but because if somebody walked out the door with a copy they would have everything. the capabilities we see at the surface are mostly the result of mining the great unknown space and attaching feeble control surfaces, ablating/lobotomizing dangerous areas, and fencing off illegal/secret/embarrassing ones.
Based off what? There’s pretty precise measurements of the capability gap where open weight models like Kimi K3 score higher than the latest flagship models just 6 months ago.
The capability gap is in the minds of the users but the frontiers' closed business models run opposite to it.
> the things are kept under air-gapped national weapons grade security measures
That's an enhancement of closed but not of moat
> because if somebody walked out the door with a copy they would have everything.
Including you and me? I'm not sure this is comforting news, despite the "trust me bro" asurances from behind the door where we can't see or verify anything.
I don't know about you but I would love to get my hands on a raw frontier base model - and a 16 node galaxy blackhole supercluster to talk to it. the capabilities currently being loboptimized for are just a narrow market-shaped slice that cheaper models can distill a competitive subset of but the untapped power of the base is a true technical moat.
about the benchmarks: the difference between a distilled competitive subset and the untapped base might be the difference that matters in any given task