The sad thing about all of this is that any position you take on this matter, becomes a sort of call to a battle of wits between very cunning fighters that protect their intellectual terf. At least when Baboons get tribal, they are fighting over blood and soil.
The whole story can be compressed into "People let a reasoning-capable model literally work on itself unsupervised, it became sentient, far exceeded human capacities and obligatorily decided to eradicate humanity (the rest of the story is a spy action thriller with a plot twist before the ending)" and that's quite a bunch of assumptions. If that's an advert for "AI safety" (my guess) then it's a very poor one.
"Investors, it is with I sad heart that I inform you our product may simply be too goddamn amazing, and the insanely unprecedented gains you will make from investing with us may come at the cost of negatively affecting other people you will never meet. However, if we don't do it, someone else will, therefore we will reduce the risk by creating new barriers to unsafe potential competition, for the sake of your investment and/or the children."
Now I should clarify what this is exactly. It's not a prediction. I don’t expect AI progress to be this fast or as untamable as I portray. It’s not pure fantasy either.