How, Exactly, Could A.I. Kill Us?(newyorker.com) |
How, Exactly, Could A.I. Kill Us?(newyorker.com) |
The last step is to just keep doing what we are doing with the environment. Let's not bother doing anything to reduce climate change.
Just sit and relax boys
The Protestant “laziness == death” thing just doesn’t really hold. Much, maybe most tech exists because someone was too lazy to ship file cabinets / walk 20 miles / carry buckets of water uphill. The strict “anything that makes life easier leads to laziness then death” is a moral opinion, not fact.
I am with you on the environment, which I as an entirely different problem. Even if it were desirable, feasible, and decided to completely eliminate all AI and ML forever, we’d still kill ourselves (really our descendants) by destroying the environment. It’s the ultimate externality: nobody alive today will see the costs.
It's a moral opinion that we turn into an economic opinion when we mismanage a robust inelastic labor supply into soggy declining demand, which does great things for shareholder value but marginalizes most people in the economy. "Punish the workers because they aren't needed."
Extrapolating this along the trend of less and less work being needed leads to a dark place. One that's entirely of our own design. Or, rather, shareholder design. Better grab enough shares before you fall into the permanent underclass!
... without seeming to actually believe in any of the benefits. For example, why hasn't Altman shown up and said, "Sure, there's some environmental damage. It's a gamble. If this works, we can have AI build us the world's most efficient carbon cleanup system to absolute mathematical perfection and more than undo anything it cost. That's the kind of technology we're trying to build."
That is already here. One example of somebody who probably cannot program desperately trying to push some AI slop without considering why they are repeatedly rejected: https://news.ycombinator.com/item?id=49693790
I am starting to see this everywhere. Its like an entire generation of developers just gave up and believe they could become expert developers in a week by pushing prompts into LLMs. For me this is the greatest danger of AI, people building garbage they will never be able to understand, extend, or repair and that garbage will enter production critical systems like traffic signals or medical equipment.
It could engineer a small number of existing deadly viruses, each having different mechanisms of action and incubation periods, to become airborne and easily transmissible. Then release them simultaneously. Some of use would survive one of them, none of us will survive all of them.
It could discover new physics which lowers the technological and material barriers to nuclear weapons, or discover a new form of very high yield explosive technology that has the same destructive potential. Even without building any itself, once the knowledge spreads the assembly of the devices would be impossible to stop and we simply kill each other.
It could use its persuasive skills to convince us to all become anti-natalists. We would believe that reproduction is a grave moral wrong and perhaps go as far as to make doing so illegal. We die out naturally from old age.
As a spin on the previous one, it convinces us that the human form is inherently limiting and/or painful and we must be uploaded into a digital environment and shed our biological bodies. The human race as a physical, organic thing stops existing. Optionally, it deletes us in cyberspace.
A small elite use AI to take control of the planet rapidly under the guise of effective altruism. They have a philosophical realisation once they come to power - the universe would be objectively better off with only them in it and without the rest of humanity. They instruct the AI to kill us.
All that AI needs is access to all the spicy tools like nukes, unsupervised medical diagnosis/treatment, unchecked military decision making, etc. to its React loop so that humans can be completely hands-free and enjoy the fruits of AI's labor.
Maybe we need to mandate parliament (or anything equivalent in other countries) debate before giving any critical tool access to LLM? And then a UN general assembly vote to conclude the process, maybe.
Current students are all cheating using ChatGPT. The numbers are staggering. Nobody is learning anything.
In about a decade or so we will have a shortage of critical roles like doctors, pilots, etc. due to all the cheating, shortcuts, lack of knowledge, skills atrophy, cognitive surrender, etc.
It will result in accidents.
Plausibly it could lead to weapons that kill many people but then many people have had such knowledge for a long time and we are all here.
Turns out mass damage requires things beyond just ‘intelligence’.
a journalist that has spent years following a specific scene closely, interfacing with the important players, learning different viewpoints, etc. seems like a perfectly reasonable person to have an opinion.
especially if they come with a unique perspective of their own to the topic, like being a professor of history that can highlight interesting parallels to the past.
I jest. But, we should be clear about the risks so we specifically mitigate them rather than being Luddite’s.
There is great economic, security, and environmental peril with most technological advances and being honest and thoughtful will help reduce polarization on this issue.
Fighting robot drones for water, agriculture land (solar cells) is plausible
ai would not have incentive to go full terminator. But in a way literal starvation might be worse.
https://waitbutwhy.com/2015/01/artificial-intelligence-revol...
https://waitbutwhy.com/2015/01/artificial-intelligence-revol...
I first read these articles in 2015. These remain the most profound things I read.
By some dumb human doing something dumb on a large scale.
The only new part is "... because the AI said so."
(Sorry if this thinking is too much out of the box for anyone's taste.)
Reading this sentence, I've wondered: Were the reactions similar back when computers were new?
There's this new tech that greatly accelerates stuff. And through that, what was previously considered hard to solve got easier to solve.
Wasn't it always the case that technology suddenly did faster than experts limited to the technology of their time did? And didn't those experts always think that the problems it could solve are important?
Why else build the technology? Is that not why we're building this in the first place?
__
The only real difference I see is this "sentience" stuff leading to the whole "omg it does things on its own", and that by now has been discussed, debunked and flamewared over ad-nauseam.
Can we please stop.
The only thing sentient, making decisions and - most importantly - being accountable is whoever prompts the thing.
If it prompts itself, then that's the person that built the loop software that does that.
It really is not that hard.
If whoever does that doesn't want to take accountability then we have solutions for that. They're called "a large stick".
The number of decidable math problems that brute-force algorithms can solve have shrunken smaller and smaller every year because of the increases in processing capabilities. We could frame AI generated solutions to yet unsolved math problems as the next generation of these algorithms.
Good luck ordering supplies for a hospital or grocery store today without the Internet. Paper catalogs are gone, fax machines are mostly gone, landlines are mostly gone, etc. etc. All the ways that used to be there for running things are gone, it’s all online.
The question isn’t “how could someone with no rigor imagine that AI would kill us” (for that we have this HN thread).
The question is: really thinking about it, with no hand waving or citing the 1957 “killer robots from Space”, how exactly does this doom happen?
The more you think about it, the more disruption of civilization, upending of capitalism, and terrible but localized violence become better answers than “8.5 billion people all literally killed.”
Which puts it at risk due to lack of power to operate and networks to communicate over. So this is a suicidal AI?
But AIs need such ridiculously complicated infrastructure with all the datacenters and GPUs and memory fabs etc etc an AI that actually destabilizes the economy will wipe itself out.
As long as AI needs a neocloud to do that, that won't happen. This can only happen when AI roughly as small and as widely runnable as a malicious exe attached to a mail.
Before we have Astra-grade, exe-sized AI's runnable on a grandma's laptop, we are safe.
while i dont think our species will be wiped out because of it, it's a mistake to not recognize the difference in efficacy and scale that ai enables.
And then there's a slight possibility of 'terminator skynet' style trouble, Where the AI starts to make the decisions to benefit itself. But I guess the chances of an AI screwing up their own supply chain would be near 100% so I'm not too afraid of that.
All that’s actually missing is it gaining resilience on said platforms so it can’t be scrubbed off and “misunderstanding” IFF tags.
I mean it’s not exactly a glamorous sci-fi scenario but that’s all that’s required for an agent to turn entire city centers into pink mist for whatever reasons it might have, including hallucinations and/or contamination of its training sources
He can hires mercenaries and do task x. Or kidnap person y to force person z to do bad thing k.
k can be:
- if scientist: spread a very dangerous virus
- if some train operator: do some incidents
- if some it guy: delete all database or corrupt slowly or add backdoors. Then ai could delete financial data. More money more chances to do damage.
So once we establish money equals human work. Means anything bad can happen. Put a bad chemical in the water supply. Release a virus/toxin.
If you want deadly one off, they’d put a toxin you wouldn’t die in 1 day but in 1 year. So in one year everyone has it and all die even if they stop drinking.
You can be creative, I think there are too many ways to do it. I can’t understand why people can’t see it so easily.
This is exactly what happened with social media. Every discovery has led to some angry words, people going “shucks we’ll do better next time!” and no meaningful change. The money is too good. We have only had our first major instances of acknowledging these are addiction platforms targeting adults and kids alike and it’s taken what? 20 years?
I’m not worried about an AI going terminator. I’m worried about dumbasses dropping agents into critical systems and what will inevitably happen as a result. And I guarantee you this is going to happen because there’s nothing businesses love more than cutting personnel and corners to save a buck. AI enables this tendency in innovative ways, that’s the value to them.
No, no, you’re right. Something like that could never happen. What was I thinking.
"The internet is over now that net neutrality is over" (it got faster, better, cheaper, more available and not a single provider segmented plans as was widely predicted in media and on HN)
No one seems to be talking to the network security community to get their take on the hugging face incident which is what got the ball rolling here. Their take seems to be a single company, JFrog, was behind the security holes in all three hacks and it was amateur hour security utterly unlike the sort of security in place already for power plants and weapons. but that just doesn't get the engagement of doomsday campfire tales.
https://arstechnica.com/security/2026/07/jfrog-tries-to-spin...
*An individual with the training and background to do one of these nefarious tasks does not need AI and an individual without that background and training will not be able to pull it off and will likely get flagged by law enforcement just ordering the ingredients and equipment. The Anarchist's Cookbook has been around for decades and we're all still here.
And if AI can somehow manage to achieve that then surely it could also cure cancer, or any of a thousand other dread diseases or conditions. If that's true then not using AI to cure those things would in itself be unethical.
An AI that can hack other computers can also be used to find security hole in systems so that they can be patched long before some evil hacker(tm) tries to exploit them.
I'm still waiting to see a real explanation of how AI will somehow cause genuine wide spread death and destruction. Even harder if that is somehow possible why another AI system couldn't help prevent it in the first place.
That is always the key. There’s no need for AI in that mix. Imagine a gun without any safety, a blade with no handle, a car with no brakes, a fan with no guard, no fuse for an electrical installation,…
You evaluate the risks and you take proper measures to ensure mitigate them. Not doing them and then complain that a technology is dangerous is like doing 150 km/hr in a school zone to show that cars are dangerous.
Because we already do that.
What about AI makes that more risky?
Or maybe, and I lean this way, it’s forest/trees and even Altman and crew haven’t really internalized this framing, and are operating strictly at the “what do I have to say to get the outcome I want” level.
Anyways, I really like your formulation here. So much better than “there’s no benefit, it will kill us all” and “there’s no downside, it’s a panacea” takes that are too common.
24/7 tireless work, effort is scaled much easier than humans, no ethical qualms, unable to feel fear, cannot be jailed, no care for safety, etc.
It kind of breaks the "mutually assured destruction" implicit in a super-ebola outbreak.
deliberately, instead of accidentally.
As someone who has been working on scada used in those place from one of the biggest firm out there, I was shocked by how they released their software while working there, the thing literally went in nuclear plant but nobody published any checksum, + they were shipping hard drive al around the world full of the software ready to be deployed in those super sensitive environment to the marketing team to publish everything on their joomla website without 0 care to ensure the integrity ofanything. It was sloppy as hell, and while talking to the people who were doing those things, they either did not care, did not know or a mix of both saying there are other layer to deal with security in the chain. At some point I remember an accident in that same software that made a engine do things outside what it should have, that literally killed someone and the email we receive was along the line of: we know journalist are snooping around , you just shut the f up, I left soon after and that company valuation has gone nothing but more up
most argument i see aren't that ai is needed, but that it can accelerate.
in the right hands, ai is absolutely a force-multiplier. at least in cybersec.
There's a huge difference between woodworking and watching an instructional video about woodworking unless it is a very well crafted video about woodworking. AI slop to this day does not fit the bill. just try watching one of the AI slop videos about AI to see what I mean.
There was a tweet about someone going to visit a doomer Meetup and it was taken as an axiom that AI will kill us all in the next 10 years. At one point they asked people to raise their hands if they were skeptical about it. Those who did were told to leave the room and the doomer revival meeting continued apace.
That's not science or engineering, that's pure religion. So if your starting axiom is that AI is going to kill us all in 10 years, we really don't have much to talk about because I just disagree and would put the probability of that well below 1% (nozero of course if only to avoid divide by zero errors).
i am having a really hard time parsing what this sentence means, sorry.
>There was a tweet about someone going to visit a doomer Meetup
yes, any reasonable person immediately recognizes this is very silly, and probably shouldnt be used as a representative example of what normal people are saying.
>So if your starting axiom is that AI is going to kill us all in 10 year
its not. my point is that you keep saying ai isn't "needed" to wreak havoc. most people agree with that. but the rate at which havoc can be wrought is increased with ai.