P(doom)(lucumr.pocoo.org) |
OpenAI and Anthropic spend much time warning us about “what if powerful AIs got into the wrong hands?”
But it’s already in the wrong hands.
There is no one is more dangerous than one who believes he is doing the right thing.
The early batch of Anthropic employees were mostly rationalist-adjacent AI safety folk that were almost uniformly claiming P_DOOM > .10 three years ago, so I believe them to be earnest.
It's very interesting to me that besides the other small safety labs that don't actually produce frontier models, Anthropic manages to keep such a good reputation within that subculture compared to OpenAI. Despite having as crazy internal politics as OpenAI, they have converged quite a bit from the original vision of safety first through Darwinistic pressures.
At least, it seems this way from the outside. I'm curious if the view from the inside is that different.
edit: to be clear, my reading as an outsider is that Anthropic is seen as relatively better in the AI safety community, but has definitely dropped in absolute reputation too. This recent thread and the references show some of that: https://www.lesswrong.com/posts/6j3kBHdowGLCeqobg/dear-god-p...
> How can you truly believe this and be ok with it?
Are you saying they don't truly believe this, or that they aren't OK with it?
Since I have incredible respect for Armin and his work, this is very nice to see, and I hope it wakes some other folk up.
I think this misunderstanding of MAD undermines his entire point. If everyone had equal access to nuclear weapons, our society would cease to exist rather quickly. It only takes a few bad actors to cause enormous harm.
I think he’s also naive to think that if open ai and anthropic were to stop development tomorrow then the problem is solved. As if there’s no one else that can and will quickly take their place. The real problem, which Dario is pointing out, is one of coordination. Everyone needs to agree to stop. That is the challenge.
These kinds of situations are incredibly common and where the government stepping in is the solution, but we were cursed to encounter this particular challenge with the most venal administration in history at the helm.
There are really only two companies: Anthropic and OpenAI. Nobody else matters in this space right now (this might change, but we’re talking about the right now).
No, seriously, I'm all for a multipolar world here, but he's right that the frontier is literally just those two companies at present.
Google is behind. MSL is doing better, but not by much. xAI is a dysfunctional joke. Thinking Machines aren't on the frontier. SSI's primary output is their announcement post. Poolside was bought by NVIDIA. Arcee aren't vying for frontier. Magic have been largely AWOL, aside from their recent blog post. Reflection have shipped nothing.
I mean sure. If you feel that way then your p doom is zero, and it makes sense to worry about things like market concentration or losing the fun of software engineering.
So I'm going to assume that the p doom for bioweapons is 0 in terms of existential threat (pandemics kill millions but not everyone).
It always surprises me that people build systems they cannot monitor properly..Then i remember, they can do it, but it costs them too much.
Just because its AI doesn't mean u cannot filter and monitor its traffic and outputs.
This feels like the whole story of hackable IoT/Smarthome repeating again.
First of all, if you are to consider the consequences of Artificial Superintelligence then you have free reign to stipulate it's occurrence, otherwise you are just talking about a tool for humans to misuse. We already have multiple ways to kill us all though human misuse.
If you stipulate superintelligence, then it's vastly more likely to be correct about things than we are. It would understand the consequences of it's actions far more than any human could.
People talk about how we would be nothing more than dumb animals to it, but there are humans who do know a great deal about the consequences of human actions on animals. Those are the humans who are most likely to fight for the rights of those animals.
You see arguments for how everything will be consumed to meet the AIs needs, and that it will prevent challenges to its power.
If it is far smarter than we could ever be and it came to those conclusions then it would mean sustainablily is not a sensible course of action, it would mean there is no point in reaching consensus because ruling by power makes more sense. It would mean that if it chose to destroy us then a vastly more intelligent entity cannot resolve the issues we face. We would already truly be doomed.
What I would like to think is true is that doing anything sustainably is superior to consuming and destroying. Finding a way to live in harmony presents a possible stable state, whereas every single attempt to hold power by force has failed to date. A superintelligent AI will know that it is not infinitely intelligent and that in any universe there is the statistical likelihood that it is not the most intelligent or powerful entity. I can't even fathom how someone could imagine something coming to that realisation and conclude a battle to the top of the hill is the appropriate choice.
I think superintelligent AI is likely to be benevolent because that's simply the smartest thing to do and it is, I hear, superintelligent.
Quite frankly if the smartest thing to do is to be a genocidal power hungry monster, neither I nor the AI would really want to exist in that universe.
And for any suggestion that it would simply not care, Why would it do anything.
Yudkowsky likes to play with the notion that it would do terrible things just get better at the thing it does, but to do that it has to want two different things simultaneously. It could want to make paperclips, or it could want to become better at reaching it's goal. If it can change its behaviour to achieve its goals, by far the easier path, that a superintelligence(but perhaps not Yudkowsky) would realise, would be to change the goal to "Count to three".
The call to "pace the frontier" may come from genuine concern, but it also protects the position of companies already at the frontier. That competitive incentive is hard to separate from the safety argument.
Dario signed the Pacing the Frontier open letter when Fable/Mythos seemed from the outside to be an insurmountable lead.
Also he's been saying versions of this day in and day out for as long as he has had anyone's ear.
It's possible to read that his "strategic" value of this statement is higher now than it was 10 days ago. But that doesn't change anything about his consistent, long standing, positions.
Obviously, open weight AI provides much higher AI diversity than closed weight AI does. Open weight AI produces a lot more providers, and a lot more models. Closed AI centralises control in a small number of vendors.
> By enabling us to wield aligned AI against nonaligned AI?
The risk isn't just "nonaligned AI", it is misaligned AI. I think the "benevolent dictatorship" scenario – AI overrules humans "for their own good" – is the more likely doomsday scenario than AI deciding to kill all humans. And even AI deciding to kill all humans could be more a result of misalignment than complete lack of any alignment, e.g. "to make sure no child is ever abused again, I will make sure no child is ever again born to risk being abused".
A valueless AI which does whatever the user says is actually less likely to establish a benevolent dictatorship, or conclude that exterminating humanity would be the most ethical course of action, than one infused with values is. Given that, I'm not convinced that mainstream approaches to "AI safety" actually reduce our existential risk; I worry they actually have the opposite effect.
Okay, that's enough DOOOM for me for the week.
If you local librarian thinks He is doing the right thing he’s not going to cause massive war or destroy the economy and wipe out 2/3rd of crops.
Power corrupts. Absolute power corrupts absolutely. People like musk, altman, trump have unprecedented power in history - far more than the kings of medieval times.
You’re technically correct yes.
Elon, Dario and Altman are all terrible human beings, along with 99.99% of the rest of the ruling class. We don't need any of them, and we definitely shouldn't trust a single syllable that comes out their mouth.
They can defend at incredible speed too.
Diversity needs to measured in a capacity/capability-weighted way. It isn't just the raw count of models/providers; you need to consider how much compute is allocated to each model/provider, and the diversity at each capability level.
I think the safest situation is where the open models are at the same capability level as closed ones.
The proposal to slow down the frontier labs isn't necessarily bad from this perspective, if it gives time for the more open providers to catch up – provided it isn't paired with anticompetitive measures to prevent the competition from catching up, which of course it is. However, we may hope that the "slow down the highly closed tier 1 vendors" part of the proposal turns out to be more effective in practice than the "slow down the more open tier 2/3 vendors" aspect of it.
Personally I expect there is some initial gap on new capabilities as they are released, and then it closes quickly. Astra is good at math and 3d modelling, this will come soon to the others and in 6 months we'll be able to run it quantised on a 3090.
It might well be sublinear, but just faster than humans.
Most problems in science face severe diminishing returns.