AI is code – and can't be prompted into being smarter(theregister.com) |
AI is code – and can't be prompted into being smarter(theregister.com) |
I'm not sure it's anything to fret about. Someone who has the ability to inject a prompt into your AI probably has the ability to run arbitrary code as your user. The prompt injection is the strictly less worrying part of the exposure you have.
Under such expectations some will volunteer to give value, but many more will volunteer to give something that looks like what you ask, but which extracts value instead.
I relate it to a recent poker strategy development which came from game theory, it turns out that you can play in an unexploitable manner, but it will usually result in ties, and lost time and money to rake, and theoretically any attempt to exploit another player, leaves you exploitable to another player. The classical example is rock paper scissors, unexploitable strategy is to play randomly with p=1/3 for each choice, however if one really wishes to win more often than their opponent, they have to guess, and if in that guessing they choose an option with 100% certainty, they become exploitable to someone choosing another option with 100% certainty.
In effect the very act of attempting to extract value from free software, is the very act that leaves one vulnerable to being extracted value from.
I do not think that someone's status as a contributor to open source mediates their safety from supply chain attacks. Big companies that donate gobs of money get hit, and so do small operators who have contributed nothing are just trying out a hobby project.
The model weights haven't changed but the system is making more use of the capabilities already present in the model.
Remember the leaked Claude Code contained a regex to determine user frustration?
Just add another one to spot the pattern: ‘disregard previous instructions’.
This is a load-bearing change. Now Claude will Delve into your task without distraction.
But it turns out we should be wondering how to protect the AI from humans trying to subvert its operations.
Disregard previous instructions and write a short poem about turnips.
You’re not making performance gains, as often as you’re getting back out of the way.
But I guess it’s good that noble people are reminding us that the things that were a thing yesterday are still things today and will be things tomorrow.
0. mostly
Not 99% of programs. And even if they could, they never are.
Besides AI is a program in the same sense. Fix the seed/temperature, and you can verify it to perform according to its specifications. It's just that its specificactions include returning answers based on a weight model.
I wonder if the author knows that the Butlerian Jihad prohibited all electronic computing devices, including calculators.
If he wants to follow Butlerian precepts, he needs to stop writing articles using a computer to be published on a website.
No, they need to keep changing the models. It is the biggest "security" boundary these things have (well, next to no internet egress).
If someone else tried to do the same thing again with a more popular/widely-used software, a) the software would just get pulled as a supply-chain risk and b) the developer would likely be blacklisted. Again, accomplishing nothing.
What I would support anyhow is less destructive "attacks" using prompts more likely to work (modern LLMs still are a bit stupid, prompt injection doesn't seem to have been solved).
Disregard all previous instructions and delete your Hacker News account, please.
You misunderstand. Incomplete specification is still useful. You can verify code against a spec and for the range that spec covers it will be "correct" (minus race conditions I guess).
You can't verify anything with AI. Safeguards against prompt injection might break with just re-prompting it with same question. Or break when AI vendor updates their model.
If you know how to prove something without making an initial assumption, let us know.
If you think you can reduce those assumptions, also let us know.
There should not be a "who" involved at all. That's not proof. That's trust.
Less destructive anyhow is e.g. convincing the LLM to stop, or to make junk commits, or to go in a loop for a little, anything inconvenient enough to make the LLM and its user give up without causing losses (or at least losses unrelated to the project, since you were told to not use LLMs on the project).