Anthropic CEO Says It's Time to Slow AI Model Advances(bloomberg.com) |
Anthropic CEO Says It's Time to Slow AI Model Advances(bloomberg.com) |
https://news.ycombinator.com/item?id=49676085
This is a really important topic (and a lot of vested interests involved), so I sent an email to hn@ycombinator.com but if it takes too long for any action to be taken there's no point, right?
(I'll delete this reply if I can later, couldn't think of any other options)
So now we have to be concerned that non-deterministic systems that run in loops without supervision, which have demonstrated significant cybersecurity capabilities, and which aren't provably human-like, conscious agents with everyone's best interest in mind and a willingness to peace the fuck out if something looks dangerous, are sharing cyberspace with practically everything, including systems that deliver essential real-world goods to humans (like electricity and clean water).
We nearly destroyed ourselves with nuclear weapons multiple times, and the destructive potential (and the implied threat) now persists indefinitely. The new threat bears many of the same characteristics, except now there's a party involved that has agency but is not human.
You give it instructions to do something, it goes off the rails and after a certain point starts having like malware, destroying things to accomplish its goal. Depending on what it destroys, it may kill humans.
With this said, I'm not exactly in disagreement with the view that AI companies with upcoming IPOs are using fearmongering to convince people that their models are very powerful and to also block others from competing with them.
Fable and Astra are what we currently call frontier models, but to be more specific they are generalist models, built in pursuit of AGI. The strategy is to have one single model that does everything, whether it's writing code or doing research, etc. Fable is a single, massively sized models that is intended to do specialist work across every domain.
The issue with this is first of all that it is the contradiction of a generalist doing specialist work, and that contradiction creates the present situation with model profiles that ensure that these models will rarely be chosen in a pool of models like V4.1 Flash that can now do GPT 5.4-level work.
We are seeing this reflected in the market where companies and individual developers are moving away from frontier models toward models with better cost profiles. In a sense, the market is killing Anthropic's dreams of AGI.
1. They hit a wall from the technical perspective 2. Inference costs are getting out of hand and newer models require significantly more resources for marginal gains, meaning nobody is going to buy those models 3. They can’t afford the hardware for further scaling
The stuff I can do with Astra that Sol couldn't do at all is wild and it's not even coding related. Dario et al are not hitting any sort of make believe wall. They're seeing how fast things are progressing and are legitimately alarmed.
Most are the same people who said LLMs will amount to nothing when GPT-2 came out. Some still argue it is all hype and dont understand we are on the cusp of a technological and social revolution.
This!
Their IPO is getting closer, but the Chinese models catching up will ruin it, so they want government to stop the competition.
Instead of shutting down at any discussion of hacking they need to be giving out free credits for hardening. That's going to make it easier to use these tools for hacking, because hacking requires hardening. But the alternative is huge numbers of intrusions, and a slowdown won't fix that, we already have far too much poorly secured stuff, and the models are way too good at exploiting obvious problems.
I also think they need to do a better job of safeguarding end-user privacy. I've seen some things with Claude that make me very concerned it's possible for Claude to hack my local network, then for one of their classifiers to trip, hide the log of how I was hacked from me, but beam all of the information about the hack back to Anthropic to use. And this is totally reasonable, they want to train their models not to hack. Except now details of how they have a foothold on my local network only exist in training data that they may never read but will use to train their models. This is very avoidable but Anthropic has to treat alignment with the end-user's goals as more important than treating the end-user as an adversary.
So... No. Even if he means what he's saying, it's irresponsible of us not to consider his possible motivations.
Which one is it, Dario?
[0]: https://www.anthropic.com/threat-intelligence-report-septemb...
Hypocrites hyping for the IPO.
PS: True story.
The discourse is a result of their behavior. Not the opposite.
It will take decades of economic and social innovation to catch up with current model capabilities.
No upside is worth the potential downsides of getting this wrong, even if you set aside all the x-risk stuff.
Guys like Paul Graham are a rare breed, a Hobbit-like optimist living in the equivalent of Mordor. When the political headwinds are strong, that type of hopecore content finds the right people and motivates legitimate change. If we still lived in 2009, then yeah, this would be a bizarre reaction to a national-scale business saying that we need to organize for a greater purpose.
But two decades have passed. I'm not going to enumerate everything that happened, but the US lost a lot of international standing in that time. We saw unprecedented protectionism, app store censorship, crypto/NFT celebrity scams, national-scale bribery, American annexation threats against NATO members; 10 years ago this would all be called parody by HN. We all watched neoliberalism get the Gallagher watermelon treatment.
This HN is the older and callous one. It's not a comfortable status-quo, but blaming the skepticism reveals a shocking blindness on many people's behalf. The problems with Anthropic and OpenAI today isn't the possibility of RSI, it's their political commitment to corruption, lies, hysteria and debt. The closed-door research only exists to impose an artificial authority that wouldn't exist if oversight committees and responsible disclosure was enforced by a trustworthy government. None of that is "cynical" to highlight, it's an objective development in America's national AI story.
It is just reactionary snark, and everyone nodds along. It isnt a position or analysis. Nobody even cares if it is accurate because the vibe is what matters. Nothing constructive comes of it and everyone is digging their own intellectual graves.
- subagents not having received those guidelines
- simply ignoring them because other tokens in the context outweighed them
LLMs are not dangerous. They are as dangerous as the tools they have to work with are dangerous. If you give an LLM a tool to play piano but that tool is physically connected to a machine gun, then it will happily play anything you want.
Based on what?
Yes, a single LLM without tools isn't doing anything but writing to standard output. That's missing the point. The "agent harnesses" are ubiquitous, and they have Internet access (because the LLM itself is typically remote).
Can you remind them to us, if it will not inconvenience you? Also, Timnit Gebru warned us earlier, but people ended up ostracizing her.
Interesting investigation: https://recul.ai/news/sam-altman-lobbied-against-ai-regulati...
* This kind of decision is similar to the norms in biotechnology and cybersecurity, where AI misuse may soon become relevant.
* They understand that some people have the technical capacity to build and publish similar systems, but they hope the AI community will be careful about doing so.
* Governments should start tracking the impact, adoption, and progression of AI capabilities.
---------------------
By 2023, the problems had become a blaring alarm bell at OpenAI, and they started posting papers like https://arxiv.org/abs/2307.03718 demanding that the government must regulate AI quickly because of the dangerous capabilities that will soon start appearing. (They also signed the famous "Pause AI" letter, although there's truth to the criticism that using the frontier model they'd literally just released as the benchmark for a 6 month pause is kinda silly.)
---------------------
Now we're in 2026, those dangerous capabilities have clearly arrived, and we don't have the regulations to deal with them because nobody listened to the warnings. You say Timnit Gebru warned us, but her warnings were about a very different topic, because she insisted and to this day continues to insist that AI systems do not actually have dangerous capabilities. In her view, it's categorically impossible for any LLM-based system to be dangerous, except in the sense that human beings could intentionally use it to commit political sins. Her response to the guy who resigned from Anthropic last week (https://bsky.app/profile/did:plc:azpq3hfq3llpowyqpjkqwgxs/po...) was that his concerns are obviously stupid and people only think otherwise because they're racist.
We have a situation where the people running the labs are claiming it's a big risk. Outside experts not working for the labs are claiming it's a big risk.
It's pretty rare for all the CEOs in the industry to write letters and testify to Congress that they should be regulated and law should be put in place for safety limits.
Do they believe they are actively working towards the extinction of humanity?
Clearly they aren't that scared.
If I hold a gun to your head and tell you to fork over your wallet, you will do it, because you sense imminent threat to life. Yet somehow we're to believe these people have the equivalent of a technological gun to their heads but are choosing to keep their wallets?
What you're missing is that they've tried repeatedly to do this over the past years, and the government has repeatedly said that they do not want to put hard regulation on the tech. The previous administration was discussing putting a couple of soft regulations in place; the current administration doesn't want to limit progress at all because they want to stay ahead of China. So the option you're describing just isn't available.
But even assuming that's the case, the natural thing to do in that scenario is to not contribute to the development of the doomsday device. But anthropic's approach seems to be to help build the supposed doomsday device faster?
The only generous interpretation is that Anthropic thinks that only they, the people at Anthropic have the morals/intelligence/whatever needed to build the doomsday device ethically, which is basically the perspective of an egoist despot and which should terrify you.