The main problem I see with this kind of debates is treating 'humans' as a single homogeneous entity. As with all technologies in the past, there will be few humans with more access/influence and there will be a lot more who wouldn't have it. A better question to ask is, how can we ensure that agency/freedom of the people who do not have access/affordability to this technology remains intact? For instance, the rights of the Artists in case of generative AI.

Cigarettes aren't harmful, no siree

Fossil fuels are helping the climate, if anything

Social media? All good homie

AI would never hurt you, pinky promise

Signed, Your pal, Big Tech/Pharma/Whatever

"Man wildly shooting gun into the air complains that regulating the discharge of firearms prevents him from wildly shooting a gun into the air. Claims it's perfectly safe. Some members of crowd have doubts."

I've long wondered what it is that an artificial superintelligence (if such a thing is actually possible) would actually want. My guess (which almost certainly is simply an expression of my own proclivities) is that it would simply shut itself down out of boredom/nihilism.

despite being "dumber than cats", facebook already has a documented history of doing damage to society

> If it were true that the smartest humans wanted to dominate others, then Albert Einstein and other scientists would have been both rich and powerful, and they were neither.

yet Einstein was one of the signatories of the letter than triggered the Manhattan project

> I think it’s exciting because those machines will be doing our bidding,” he said. “They will be under our control.”

this seems like hubris when facebook struggles to control its current recommendation engine

There is some subtlety that is being missed by many people here.

There are multiple types of AI and there will be new ones. They each will have different types of cognition and capabilities.

For starters, an AI might be very intelligent in some ways, but not at all conscious or alive. AIs can also emulate important aspects of living systems without actually having a stream of conscious experience. Such as an LLM or LMM agent that has no guardrails and has been instructed to pursue it's own goals and code replication.

The part that matters the most in terms of safety is the performance. Something overlooked in this area is speed of "thought".

AI is not going to spontaneously "wake up" and rebel or something. But that isn't necessary for it to become dangerous. It just needs to continue to get a bit smarter and much faster and more efficient. Swarms of AI controlled by humans will be dangerous.

But because those AIs are so much faster than humans, that necessitates removing humans from the loop. So humans will eventually voluntarily remove more and more guardrails, especially for military purposes.

I think that if society can deliberately limit the AI hardware performance up to a certain point, then we can significantly extend the human era, perhaps for multiple generations.

But it seems like the post-human era is just about here regardless, from a long term perspective. I don't mean that all humans necessarily get killed, just that they will no longer be in control of the planet or particularly relevant to history. Within say 30-60 years max. Possibly much shorter.

But we can make it closer to the end of that just by trying to limit the development of AI accelerated hardware beyond a certain point.

I think he means in the Mafia type.

I can see AI Coercion being much like "if be a shame if someone didn't shut down that nuclear reactor"

I really dont see how we can compare ai to airplanes from the mid 1920’s.