I have been getting flagged or near-flagged on many of my comments lately, so I will keep it short.
Approximately a year ago, perhaps closer to two, I posted an Ask HN[0].
You can either review the post, or take my word for it, but I was torn a new brown-eye for it, demeaned and even insulted for daring the ludicrous notion that such an absurdity was worth wondering about.
Fast forward to 2026, and one might now be insulted for wondering why it isn't plausible, if not inevitable to some extent.
I'd say a lot more, but I'm not in a fighting mood this evening. I'll just say that many of the folks here dismissing AI in the context of the "singularity", well, they tend to be intelligent, often technically informed, and with a fair number of karma tokens to play with. But I will tell you they are wrong. The only thing that will prevent, or defer what you are alluding to, would be a consensual pause or restriction in development. China has, thus far, made that a tricky proposition despite the efforts of Altman and Amodei et al, but consider that even if they manage to wangle a moratorium, development will absolutely not pause on the Pentagon side. Maybe the public side will slow down, but AI as 'defense', or as I prefer, weapon, will not pause for a nano second. And it will reach well beyond the extremes you suggest. And I am not sure there is anything one can do to prepare for that. My greatest concern is institutional capture, which we are already seeing.
0. https://news.ycombinator.com/item?id=32534550
Edit: All four frontier models have Pentagon contracts. Remember, Claude (Anthropic) assisted with operation Abduct Maduro, and works with Palantir. This is the direction things are moving. Just keep an eye on it, and read between the lines. I predicted this two years ago, and here it is. AI is not just about coding, or answering whimsical questions anymore.