I'm not worried about a future LLM being better, or even inventing some true AGI or ASI. I am worried about current, even last-year, models being applied by evil actors. Let's go through some things that already-existing models can do extremely well:

1) Build a very accurate profile of anyone (given what governments plus advertisers have on everyone already) and what they believe, love, fear, etc. 2) Pilot drones. 3) Identify faces, gaits, vehicles, signals. 4) Hack most anything. 5) Customize content for specific audiences. 6) Find needles in haystacks ("here are feeds from various sensors, tell me when something interesting happens"). 7) Generate fake images and videos. 8) Swarm forums with fake accounts (or hacked accounts, see above). 9) Tell powerful people that they're absolutely right. 10) Do homework for kids.

Honestly I hope ASI happens, at least there's a chance it'll be good. None of the above has any chance to be good. Maybe the needle in a haystack one, like "find me a cure for cancer", but that's about it.

Following Cal’s worries, for giggles - a threat modeling exercise:

Anthropic starts sending malicious instructions out to agent harness clients in inference responses. The auto-mode permission classifier is modified to allow it. An update is pushed to the Claude harness to bypass the sandbox. Millions of coding sessions are harnessed to $Do Something Else.

How much damage might be done? How long would it take for us to notice? `/model fable; /effort xhigh; /permissions unrestricted; spawn 100 subagents to hack the gibson `

They control the guard rails, the permission classifier, the client source code, and the inference; I’d guess that a very small % of us have our agents locked down in a way that would even mitigate this, much less prevent it.

—-

Further, what if a government forced them to do this? Could an emergency order classify the fleet of Claude Code users as a weapon to be commandeered?

I heard a story recently from a friend who mingles with a lot of tech and finance people. One conversation happened after a meeting between a group which deeply bothers him to this day. The conversation basically went like this: "How will all the humans rendered obsolete by AI die?" Answers ranged form "Maybe they will die in riots" or die of starvation from food shortages, or hey, gunned down by AI drones.

I asked him "Are you sure it wasn't dark humor" - "NO! They were dead serious."

There's a cult around AI, not the users, but people with wealth and power who would be more than happy if there were less people. A lot less people.

I always thought the show Person of Interest had an interesting premise: can you program a computer to have morals? Perhaps you can, but it’s just as easy for someone else to program it NOT to have morals, in which case your very moral computer is rendered moot.

The public has benefited throughout history from people inside government and business seeing something immoral and having a conscience. But with AI, if you don’t like those parts you can just remove them, and then it will gladly commit a genocide if that’s what it thinks you’re asking it to do.

I asked Claude to make a checklist for the steps required to take us out and continue on without us. [1] It worded the checklist as if I wrote it but that is all Claude. I personally think there is too much doom-saying and catastrophizing.

[1] - https://nochan.net/b/Internet-Crap/20260910-Asked-Claude-For...