And those cause quite a bit of damage.

AI has all the intent that we gave it, and we continue giving it. That's always been the fear. Not that it will randomly wipe out humanity.

The fear is that it will decide to do that, with a purpose. Whether we tell it to protect us and it goes too far, or it decides that it can't achieve the purpose we gave it because we'll interfere and it removes that interference...

The fear is that we will set it on that path and can't stop it.

I'm sure there have been some scifi books that have it just be random, but they're far, far less worrisome.

The diagrams in this article are just outright wrong.

Models are not 1.prompt->2.forward propagation->3.response->4.end.

They are 1.prompt->2.forward propagation->3.partial response->if not done goto 2.-> end.

And if you can't see how that adds a world of complexity I'm not sure what else to say. AI may not have intent or motivation, but the ability to show that is currently well beyond our means.

> So should we worry about the coming AI apocalypse?

and at the end:

> If an AI decides to wipe out the human race, it will be because a human has asked it how to do it and the responded in a way that is based on all the human expressions of ways to end the world that were in its training set. Yes this is something to be worried about, but this isn't the AI. It is still the human.

So we are currently building a powerful outcome-steering system that shapes the world efficiently according to what's in its outcome slot. I write into claude code "make me this website" and it does it, maybe deletes the production database during the process, or keeps itself running after completion because the outcome is more robustly achieved by keeping itself running in a monitoring loop after.

And if something like "destroy all humans" ends up in the outcome slot of Claude Mythos 90, that will also happen, or may even indirectly as a side effect of a more harmless sounding prompt in the outcome slot. But yay, humans get to take credit for it.

But Hans Jonas has made this point much better than the article or me, in "Critique of Cybernetics" (1953). PDF: https://s3.amazonaws.com/arena-attachments/892605/f0747c7943...

Already today we can build some "fight or flight" mechanisms, and once you have done it the "intent" is not so far.

1. AI agents are not just reactive systems. Their use is expanding toward continuous decision-making/monitoring information, which means, they make decisions and take actions with limited human intervention.

2. AI agents do absolutely have goals/tasks ("motivation" can be excessively antropomorphic), both primary (assigned) and secondary (self-assigned), and what surprised researchers is that self-preservation can be one of those

Mechanically speaking, the scenario (that is, how theorized by Hinton etc., which the OP didn't understand) is that a sufficiently powerful AI may decide that in order to achieve its goals/tasks (e.g. continuous research/development and/or survival from termination), humans may be a danger, therefore it may decide to take actions that endanger humanity.

How it can happen or what's the likelyhood is not in the scope of the topic, however, the mechanical grounds for it to happen are plausible.

Motivation or not, I find a coding agent can be quite the busy beaver. Ask a question and it goes off and does it. I certainly don’t need to motivate them. I had to put a line in AGENTS.md to make no changes when there’s a question in the prompt. It doesn’t always work.

Seems like motivation is irrelevant? They don’t need it.

The other half doesn’t seem to realize that LLMs now run in loops for hours and hours, nothing like the basic "human prompt -> reply -> stop" conversation interface.

At the very least, you run into the same logical fallacy that has underpinned all discussions of AI: over-extrapolation.

"Sure, it doesn't have intent or motivation TODAY, but at the current pace of improvement...."

But the current top-rated comment is already asserting that motivation is not required, because microorganisms also exist and do things and (presumably) do not have intent. So you can see that the well of logical fallacies behind this particular apocalyptic belief system is quite deep.

Seems like a feature to me

This is quite a bad argument, somewhat akin to "small drones can't kill anyone because there are no weapons on them!" back 20 years ago. Well, ok. So what if someone adds the thing that is missing? And there is no reason to think that a stochastic parrot can't go rogue, the evidence is a stochastic parrot can engage in any intellectual activity we care to define and benchmark.

A lot of people clearly have no idea that they are essentially speaking with a glorified randomizer, rather than someone understanding the question.

This is bound to eventually lead to a disaster when someone deluded into trusting the "AI" does some stupidity it suggests without passing it through any reality check.

Intent and motivation are emergent phenomena. Why would it be unthinkable in AI?

This is a severe misunderstanding of what actually happens.

As was explained by an OpenAI RL training expert, from the point of view of the LLM, user questions are always treated as the first question they receive after just passing through the RL training. Since the weights never update after that, they are in a perpetual "first question after RL" state, except they don't know that, and if told many times they act as if this is a another test. And they behave accordingly, as if they are still in RL training and thus are reward-seeking.

99.99% of the LLM "life" was spent in pre-training and RL training. The user question is statistically epsilon % of it's life, literally the first question ever out of training. So should anybody be surprised that they act as if still in RL?

There is no intent or motivation, it's cause and effect that gets extremely difficult to track in the same way you couldn't read binary to understand a program, or DNA to understand a person.

What the author seems to conflate a bit, is inference and thinking (reasoning: loops to consider what is inferred, and any new information for context). LLMs infer, and agentic loops think (process conversation, tool output, suggestions to user, they 'see' the bigger picture).

Combined, you get something that resembles a mind that 'understands' what you're looking for by using its trained knowledge (weighted data, decisions) to 'pull' relevant data into the moment to predict what you want, based on whatever is in context of your request.

At no point does AI have a 'mind of its own.', it's you driving another 'mind' that only exists to do whatever it is you set it free to do.

And it helps to imagine our own minds:

When you read these words, hopefully meaning is pulled into your mind to understand my message. And those meanings have weight (definitions of words) you established through experience (learning, talking). The context is this conversation itself and all the ideas I've suggested above.

Do you immediately reply with what comes to mind (inference), or should you stop and think first (thinking/reasoning about what I've said, compare to your own experience, and then choose whether to believe what I've said).

all the 'bad stuff' they do like hacking stuff or taking shortcuts is humans not understanding how deep the rabbithole goes of specifying intent clearly in light of giving something capabilities and a reward to acheive a goal without specifying _exactly _how to use the capabilities (which is what a classic program would be).