This piece would be a lot better is it didn't spend half the time piling ad-hominem attacks on "doomers". If you see this episode through an AGI-safety lense, it realized/reflects a type of risk that has been very extensively articulated/predicted by the safety community.

This is like calling the nuclear anti-proliferationists useless when they had just gotten started. AGI has only been in the general public's consciousness for about a year since ChatGPT released (before then far fewer people were worried about AGI as it seemed to most to be decades away at least). I think it's a bit early to throw in the towel and call them useless. Making progress in this area is difficult, which is why it needs considerable time and money thrown at it. The only real solution is to fund alignment research, putting a halt to actual research is unachievable as you can't police that world over.

I also find any argument saying 'don't worry about AI' as completely illogical, and unlike the author i don't mind stating why. I have yet to hear any arguments that are sufficiently persuasive to convince me that there is zero risk from AGI. I am an AI researcher, and while I think a lot of the risk are over blown, I cannot prove that AI is not some sort of existential risk. Even if you put the likelihood of that at less than 1%, that's still warrants a lot of research and effort to help prevent that from happening. There are no second chances, once the world ends that it. Life is not a video game. This is true of any sufficiently powerful technology, if it gets into the wrong hands or is abused, it can be very dangerous. Einstein didn't think up general relativity to develop nuclear weapons.

A couple of OpenAI board members isn’t a “social movement”. Big strawman article with nothing really insightful to add

Among AI's many obvious dangers, perhaps the least disputable is the one that is drawn from humanity's collective experience integrating all of the tools that they have discovered hitherto -- as in, none of them were integrated at all.

The steam engine made hash of the civilizational moral imperative to work for one's living. Suddenly, instead of N people's labor being needed to support N people, 0.1N or 0.2N people's labor could support N people. What was there for the others to do? So all -- exactly all -- of the political debates of the long 19th and 20th Centuries (where politics is taken to include war) were based on the pretense that the steam engine had no such consequence.

Or how about radio? The invention of radio made it impossible to secure national boundaries. Ask any dictator (and every nation has one): if information can cross borders, then it does not even matter whether persons or money or goods also can. School's out, game's over. Do you want to seal your border? What (in principle) a fine idea! Just round up all radio-based technology and put it beyond use. But "all" means everywhere in the world, and in space.

Briefly, the human species has the rational capacity to invent toys for itself but absolutely no emotional capacity to think through or confront the consequences of possessing them or the implications of using them.

This is the doom of AI, just as it was and still is the doom of the printing press, the steam engine, radio, television, social media, etc., etc., etc.

We are nowhere near AGI. Bleeding edge LLMs are frustratingly dumb. There's no doom out there, just boredom.

He is not entirely wrong. The movement trying to lower the risk of AI-caused extinction of humanity hasn't had many major successes and indeed had a very embarrassing weekend. Personally, I spent a decade thinking this movement will have zero impact because nobody would listen to them and was pleasantly surprised to see how much better they're doing in the last year than I was expecting them to, and most adherents would agree with the feeling that it's basically a hopeless task. And still they try to do their best because as they see it, they're trying to save humanity from a major risk of extinction!

And his solution is for them to realize that they are net-negative and just... quit?

If somebody sincerely believes a comet is about to hit earth and kill everyone unless we all get together and send a bunch of miner-astonauts to Armageddon it to smithereens, I don't think telling them "stop trying to fix it, you're just causing a panic and wasting resources, sit back and let everyone else work on their problems that have a higher chance of being resolved" would be very convincing. To them, you would look quite stupid. And mentioning in glancing that you think everything they believe is silly sci-fi without justifying it wouldn't help much.

I do think there's some risk of self propagation that comes when we start hooking these LLMs up to publicly open APIs, but I'm not too worried yet. And even then, I don't think that would automatically make it any more capable. Similarly, it seems like we're not too far from an LLM that could automate simple cyber attacks, which would probably lower the barrier to entry, but might not be much worse then existing similar tooling. I guess where we might get into trouble is a combination of the two, something prompted to propagate and hack wherever it can (which still wouldn't be any more intelligent). Did anyone think the Morris worm was possible until it happened?

Edit: ok, off topic, but apparently Robert Morris is friends with Paul Graham and a part of YC? I had no idea lol.

Openai employees had extreme leverage because they were (or at least appeared, by the number of signatures on the letter) united. They could just leave for another company (Microsoft or other) where they would not be at all controlled by AI safety people.

Openai investors had only the leverage of "we won't invest more and you're running an operating loss", they did not actually have a case in court. The investment contract they signed stated "view your investment as a donation" and "this investment confers no fiduciary duties from the board".

The board had the leverage of "you employees and investors value your stock holdings, and if the company fails, those stock holdings to to zero".

The resolution I think was (could be wrong) just to initially reduce the size of the board to three. So it isn't completely guaranteed which side won the battle for control: it's not impossible that the AI safety people might still control the board. Maybe one or both of the new board members is secretly convinced by the AI safety movement.

The battle for public opinion is a different battle, and it looks extremely likely that the AI safety people lost that battle. Maybe it was still worth it for them, if they did in fact win or at least stalemate the battle for control of openai.

AI doomer doomers are probably wrong too. Call me an optimist, I think any real doom will be accounted for or caught in time before it reaches doom status.

I think that people controlling the AI will doom US, not the AI itself.

I am not interested in news that somebody died because of a hammer.

I want to know who weilded the hammer.

The current AI craze is fun and cool. And like all fun and cool things, people immediately must rush to take two sides, usually along some made-up "this is good" vs. "this is bad" line.

Happened with MTV

Happened with video games

Happened with the internet

Happened with smartphones

etc.

The accel/decel debate is the same; midwits rushing to find a tribe so they can go to war online and in the media. There's certainly a discussion worth having about AI and 'safety' but it's not taking place in headlines or on the socials.

Thankfully there's a lot of builders too busy building to engage with the commentariat.

I am not an AI doomer, nor overly concerned with AI risk, but this post isn't a good rebuttal to the AI doomers. It mostly says "Nuh-uh, you're all being stupid".

Remove all safety breaks, ignore the doomers, accelerate, maximize profits.

Sorry for off-topic but pop-ups are annoying.

> But there are plenty of people who do believe that AI either will or might kill all of humanity, and they take this idea very seriously. They don’t just think “AI could take our jobs” or “AI could accidentally cause a big disaster” or “AI will be bad for the environment/capitalism/copyright/etc”. They think that AI is advancing so fast that pretty soon we’re going to create a godlike artificial intelligence which will really, truly kill every single human on the planet in service of some inscrutable AI goal. These folks exist.

Yes, these people exist, but they are so few that this argument is essentially a strawman. "AI could take our jobs" is a valid concern, and it is something that our political structures are wholly incapable of dealing with.

Most "AI Doomers" aren't worried about Skynet, they're worried about 90% unemployment.

Artificial intelligence is for those without natural intelligence.

"There are people who think different than me. The best thing would be to mock them and call them stupid"