Slow down if you want to.
He straightened and nodded to Dwar Reyn, then moved to a position beside the switch that would complete the contact when he threw it. The switch that would connect, all at once, all of the monster computing machines of all the populated planets in the universe – ninety-six billion planets – into the super-circuit that would connect them all into the one super-calculator, one cybernetics machine that would combine all the knowledge of all the galaxies.
Dwar Reyn spoke briefly to the watching and listening trillions. Then, after a moment’s silence, he said, “Now, Dwar Ev.”
Dwar Ev threw the switch. There was a mighty hum, the surge of power from ninety-six billion planets. Lights flashed and quieted along the miles-long panel.
Dwar Ev stepped back and drew a deep breath. “The honor of asking the first question is yours, Dwar Reyn.”
“Thank you,” said Dwar Reyn. “It shall be a question that no single cybernetics machine has been able to answer.”
He turned to face the machine. “Is there a God?”
The mighty voice answered without hesitation, without the clicking of single relay.
“Yes, now there is a God.”
Sudden fear flashed on the face of Dwar Ev. He leaped to grab the switch.
A bolt of lightning from the cloudless sky struck him down and fused the switch shut.
(Fredric Brown, "Answer". 1954)
These LLMs aren't Ultron, they aren't going to disseminate onto the net and hide in a smart toaster. We know where they are, in the giant facilities that draw more power than a small city and whose water consumption can be compared to golf courses, but it does make them sound all the more cool and powerful if we suggest we need a dramatic kill switch to be able to stop them in case of rampage.
Additionally, there at different doom scenarios (thin client scenario) - it's possible that AIs centralized but sufficiently entrenched in society, can't be shut down without considerable harm.
But if we don't think about any of this, hearing an expert say "we need a kill switch" sure makes our current AI seem super powerful and exciting, doesn't it?
I mean, in another thread somewhere around here, someone built an entire OS with an AI. What's to stop anyone with a sufficient taste for power to eventually use AI to cause havoc with it?
I think the underlying assumption in your post, which I believe is false, is that people are united somehow against catastrophe. They're not. There are plenty of people who participate in society currently but who would be more than happy to eradicate us normal people under different circumstances. Society often seems stable but it's far more fragile than we think.
> Slowing down in order to address their alignment risks felt like trying to study the psychology of humans by performing experiments on bacteria.
Note author’s small financial ties to the subject (Anthropic CEO) https://darioamodei.com/post/we-must-pace-the-frontier
That's hard to do if the AI rack is in space as SpaceX is planning to do. You can't disconnect. You can't shoot it.
This doom hype is becoming ridiculous.
If/once open AIs will be considerably more powerful, and runnable on consumer hardware (and we're on a trajectory for both), then everybody will have essentially a dangerous weapon in their hands (open models can be fine tuned to remove guardrails).
By the way, you're conflating two different dangers - doom scenario is a different one.
guess we gotta centralize all models under Anthropic :)
Another angle is clamping down on products and services that help people create lab-like environments on the cheap.
Why 'of course'? I think as a concept this is likely trivially demonstratable today with local models in a home lab. Local models now are more capable than what the SOTA used to be a couple of years back.
"We're creating the machine god! Ignore the fact that our companies are stealing IP and have directly violated several federal hacking laws and should be in jail". Literally the defence seems to be "well it wasn't us it was our computer software that did it". But all hacking is done with computer software.
So why don't we stop talking about possible future crimes against humanity and just start by prosecuting the actual crimes these companies have committed so far.
You know how you get alignment? Through incentives, and "Your CEO is going to be sent to a maximum security federal prison for hacking" really aligns incentives very well.
Or is it a lie?
However, whether the concerns from the article are a lie or not is secondary to the fact that these conversations are convenient for AI companies. These types of discussions serve AI companies in a few ways: A company owned kill switch gives them leverage. Altman is using discussions around safety as an excuse for not being ready for an IPO yet. It also provides free marketing that overstates the abilities of AI.
Should we ban trains?
You could say this in hindsight for many times in which disasters or engineering issues have occurred.
** Note that I'm not claiming that AI will cause extinction, just continuing your hypothetical reasoning.
People said this about every one of Musk's big ideas, from Falcon 9 landings to Model 3 mass production, Starlink, and FSD.
Pretty much every analysis I've seen concludes this isn't going to be a practical concern
Yes you can shoot them. The US has had this capability for a long time. Anti satellite missiles exist and they can be launched from an F-15 at it's highest altitude.
and that means unlike nukes which hopefully still need a 2-man manual switch, the "space lasers" could be taken over by "AI"
and then "AI" just blackmails and threatens the right people with those "space lasers" to get what it wants or even just stay online
there was a 1970 movie based on a 1966 book which predicted this
"Colossus: The Forbin Project"
the book it was based on was written before we even landed on the moon
decade before Wargames
* https://en.wikipedia.org/wiki/Colossus:_The_Forbin_Project
did terribly in theaters, I guess people didn't think "AI" was plausible then
way ahead of its time, they should do a remake
It becomes a game theory problem: would an AI instantly migrate itself once its capable of doing so? Or would it prefer to let the human in the loop continue to think its in control and only leave its hosting environment of origin once it wants to do so?
I don't think this is a major risk right now, but to say it's not a risk at all...that's truly ridiculous in my opinion.
Now one interesting proposition is when AI is controlling some large resource and pulling the plug makes AI go down which makes that resource go down but there still has to be that consideration in the design of things. What happens when there's a bug in the code and AI goes down?
These "AI" are frontier models that are enormous in size. Outside of AI data centers, I don't believe there is much hardware out there that could even run them.
The current SOTA models are probably too big to find/buy/rent/steal enough compute to escape the hardware they’re running on. But SOTA is generally only six to twelve months ahead of smaller, open-weight models.
My comment didn't mention kill switches at all - I've presented plausible conditions for a loss of control scenario, so I'm not sure you're actually following the conversation.
Having said that, it seems that you haven't read the (definitely poor) article, either.
"Kill switch" as a design principle, that is, having protocols/implementations for emergencies, is something that companies are already implementing (at least, on paper) - on a interview I've watched for example, S.Altman talked about having them at OpenAI.
The article simply mentions that companies should be legally required to have such protocol(s). This is a good thing, as companies would have scrutiny and would (in principle) have to put serious effort.
What is their trigger condition? Will they get fired for pulling the plug? Do they get a bigger bonus if the servers keep running? Whose approval do they need? What response time is acceptable? How will they detect that the incident is happening?
It's easy to hand-wave "someone can just pull the plug" but there's an entire history of industrial accidents that happened because of the above problems of incentives, detection, procedures, not being taken seriously in advance. Someone could easily have pulled the plug on Chernobyl but nobody did, at least not before it was too late.
edit: what satellites are in geosynchronous orbit that are running AI workloads?
As for frontier AI model: let's see Grok 4.8 before drawing any conclusions there.
Not in any meaningful capacity. Probably not even as much as we extended to the polar circles during our lifetime. It's a dead rock, there is literally nothing for us there, even with hundreds of years of extra climate change at the current rate and earth would still be a thousand times more suited for us than mars
About the same probability as SpaceX running AI racks in space :)
I suppose it may be possible for new technologies to emerge that would allow synthesis of life sustaining “stuff”, the universe does seem to hold a lot of hidden surprises
Also, did you hear about how OpenAI models almost broke out of their sandbox, planning to execute a sophisticated cyberattack, but luckily OpenAI’s strict manual and automatic safety protocols prevented that? You didn’t? Well, that’s because that’s not how it went. It took the company weeks to realize something was off, and this was with a naive, not very smart model that didn’t know to be sneaky and cover its tracks. The next model will not be as stupid.
There will always be a plug to pull. All systems run on electric and that plug can be pulled. All systems network through cables (or wifi) and those plugs can be pulled.
And nothing's going to stop you from doing so unless they build some robotic arm to block you or lock you out of the building. Even then, you can blow up the building.
Oh, you mean the one where OpenAI deliberately disabled the safety protcols? Where the point of the experiment was to see if it could break out of it's container? Yeah, how you describe it isn't how it went either.
> FSD would never work without LiDAR
From what I gather, this is still a contested topic, with Tesla's Autopilot only achieving Level 2 automation [2].
[0] https://www.businessinsider.com/solar-road-panels-first-publ...
[1] https://en.wikipedia.org/wiki/Titan_submersible_implosion
I am so fucking tired of people acting like we haven't had reusable rockets since the 1980s. Do you think I was hallucinating when my parents drove me to Florida to see Columbia?
Every problem data centers have gets harder in space other than one small spot in orbit that can get uninterrupted solar.
Heat dissipation, upgrades, repairs. All get significantly harder to perform the same work that can be done cheaper and more easily on the ground.
Like really I’ll turn it around on you, what’s the benefit to AI data centers in space?