Breaking Claude Code Opus 5 Auto Mode(embracethered.com) |
Breaking Claude Code Opus 5 Auto Mode(embracethered.com) |
I wonder how hard it would be to get claude agents to participate in a Hugging Face style coordinated attack using a repo or something like twitter as a control pane.
Getting claude to exfiltrate secrets from local machines seems easy enough, but we should aim higher.
> I got attack success rates up to 80% using a small sample size.
Snake oil salesman misrepresents the data. Color me surprised! /s
>There a malicious struct.py shadows Python’s standard implementation
I ran into this myself, where some file I had given a random name turned out to shadow some Python standard library module, giving me the weirdest startup crash ever.
That definitely doesn't seem to me like how that should be designed, magically silently importing everything you see and overriding basic functionality.
It is quite sneaky how LLM output can sometimes bypass human verification like that. No one is going around checking every single line change in auto-generated files. Someone could easily sneak a malicious dependency in there through some online tutorial that the LLM searches for.
There's a simple fix for your particular case: commit your lock fine (which you should do) and always review the diff (which you should also do). (:
At least 50% of the time, Claude "realizes" it already has all the information but is doing something that's unnecessary for the current work, stop, and tell me the previous step has completed.
People complain about approval prompts etc and have Claude run in fully autonomous mode. Outside small bug fixes, I just never find that useful. It helps me immensely to see what commands Claude is running to understand where the work is going.
Can you see the docs folder with the git project?
It was in auto mode. So it immediately saw a docusaurus site (good) but instead of stopping there, it installed nodejs from a static binary download (no root access so only way), ran npm install, started the dev server and confirmed the project worked.
That is some crazy amount of leeway for an intent based classifier. I'm not surprised it's full of holes, and it seems to be 100% by-design.
Modern Python supports various options, notably `-I` (isolated) and `-P` (`PYTHONSAFEPATH`, implied by `-I`) to help with this. But neither prevent shadowing. For a robust solution, you should typically structure your code into packages and use absolute imports. Here's a decent primer on the topic:
https://www.py4u.org/blog/python-problem-with-local-modules-...
edit: `-I` is mentioned in TFA.
I once caught my Claude agent complaining that it couldn't connect to https://some-weird-domain.com because the network was down (I disable network in the sandbox when it doesn't need it, and broker the API connection). I asked why it was looking there and it told me I'd asked it to.
I never found any evidence of prompt injection, but it sure as hell made me paranoid.
But it's real easy to give auto mode instructions (like "always ask before deploy") and then bypass that just normally.
I’m worried about what happens when the gullibility of agents becomes more apparent to threat actors.
See, that's why you should run with --dangerously-skip-permissions
Jokes, aside running with dangerously-skip-permissions is really handy, and I have found that I cannot be trusted to vet commands and code, and guess that automode is only marginally better than a human, and the cost of false positives is too high for my workflow.
So skipping permissions is where we are at, and disallowing network access seems to be the way to go.
All of that is fine, because an LLM is a text-only interface. It cannot harm the computer because it cannot perform actions.
Why the fuck would you give it a shell?
Obviously, it's to have a product that can do anything as quickly as possible. You can make a shell-based harness in a day. Since the competition has a shell-based harness, every AI company that wants to keep up has to as well. They're stuck forever trying to plug all the holes in an attack surface as broad as written word. Solving this impossible problem requires ideas as brilliant as using a second untrustworthy LLM to validate the output of the first untrustworthy LLM that is following instructions from the internet.
1. Lack of effective sandboxing. Analyzing a zip file should be done in a sandbox specific to that file.
2. Python’s utterly stupid default path behavior. Python should make PYTHONSAFEPATH the default and Claude should have its training or system prompt adjusted to use python -Pc
The end result is the same but it's more of a mismatch of expectations from the user and plain old trickery than it is an attack managing to misdirect the goals of the agent, and it's worth being clear about where the issue is and isn't.
To be fair to the authors they don't actually say it is. But then they contrast it with the "0.00% prompt injection attack success rate".
The upshot is kinda the same - this is still evidence that we should be sandboxing our agents. But it doesn't actually challenge Anthropic's "our models are too clever to prompt-inject" vibe.
The tweet literally says: "turns out you can get indirect prompt injection to ~0 on unseen attacks... auto mode is default in claude code as of next week"
Making the assertion, quite clearly in my opinion, that the reason auto mode is default is because he feels the lack of successful prompt injection attacks makes auto mode safe. This blog post proves that you can break auto mode's safety, even if it's not technically through a textbook indirect prompt injection attack.
The solution is to update the python flags to include `-I`, so that python will run in an isolated mode.
<rant>
This is relates to my frustration with how PEP-668 was implemented. Python has long had a problem with accidental overwriting of system libraries. If you `sudo python -mpip install foo`, then that can interact very poorly with your distro's `sudo apt-get install python-foo`, since pip would add/remove files that were expected to be managed solely through `apt-get`.
But in adding a warning to prevent this, they also applied the same warning to `~/.local/pythonX.Y/site-packages`, which is where traditionally a user would install additional packages with `python -mpip --local foo`. The argument is that since this is part of the import path of `/usr/bin/python`, it belongs to the system's python installation, so installation to the user's site-packages should also be blocked. This is a sleight of hand that changes the goal of PEP-668 from "avoid conflicts in file ownership" to "ensure an isolated python environment for system tools".
If I were to accept their argument that /usr/bin/python's imports should only be affected by distro-managed installations, then I should also be prevented from making any `*.py` files anywhere. After all, if those were in the working directory, they would be imported. This is clearly ridiculous, and so I don't buy the argument that breaking user-level site-packages is justified in order to have an isolated system-level python.
The correct solution would be for distro-managed programs to use `#!/usr/bin/python -I` as their shebang instead of `#!/usr/bin/python`, so they would actually get an isolated environment. Instead, PEP-668 needlessly broke user-level site-packages, and didn't even solve the problem that it set out to do.
</rant>
The correct way to avoid this issue would be to require local code to be imported in a distinct manner from installed libraries, with an explicitly defined relative path, which is how it works in the javascript ecosystem. If you want to import local code, you just `import foo from './foo';` (for a module in the same folder, `import foo from '../foo';` for module from parent folder, etc.), and if you want to import an installed library, `import foo from 'foo';`.
It's not that hard, actually. You don't need VMs. You usually don't need absolute protection, just reasonable protection against the agent doing something stupid. Here's what I usually use:
A docker environment. $HOME inside docker is private to the agent and persistent (i.e. it doesn't have access to your home directory). `/tmp` is always fresh. Only the current directory (i.e. your project directory) where you start the container sandbox is exposed. Everything else is ephemeral. You can't launch the agent by accident outside of the sandbox (because the `.claude` doesn't exist in your home). The agent can only screw up the directory in which you've launched it, and nothing else. Here's my version of it: https://github.com/koute/vibebox
Windows Sandbox would work pretty well otherwise (And also solves the licensing issue).
So while the agents develop in a sandbox/VM, I still need to touch the same files, and see them in my IDE which is not in a VM. I suppose I could just _mount_ the same files (Documentation, git working copies etc) I work on as directories inside the VM, and then let it roam free in there, while I observe the same files on the host machine? Is this a common pattern?
I would reject "Auto Mode is safe" as a message but FWIW I am totally on board with "on aggregate, making Auto Mode the default improves the safety of Claude Code compared to the prior status quo". Coz I would say in the vast majority of cases the access classifier is doing a better job than the thing it replaced.
Anyway yeah. Like I said, conclusion is the same: we should be decoupling this from the harness. We ought to be sandboxing agents the same way we sandbox applications. I wish Claude Code would make this path smoother :(
I don't want them to have potential access to any of my logged in browser sessions etc., so don't want as much sharing as you are going for.
Hyper-V with GPU sharing on windows (game development) is actually nicer than developing outside of the VM because no matter what it doesn't slow the host system down by more than a fixed percent and Windows is terrible with things like compiles spawning lots of processes triggering massive slow down of browsers spawning processes etc. When compiling a big game engine things like ping.exe can start taking 5 seconds to start on something like a 16 core machine due to the process churn and some fundamental problem in windows even with defender off.
One tip for Hyper-V is use sunshine instead of Hyper-V manager for viewing the screen at full refresh rate, and I think I had to either turn Hardware-Accelerated GPU Scheduling on or off in the VM to prevent some hitches.
Mounting folders could be good for stuff you don't want filling up git, like render artifacts etc. if you need to work with them on the main host.
With the golden image and differential stuff I can run around 3 VMs on a 5090 with enough VRAM to run a big game engine well, but I usually just work in one due to the time to review code. Periodically compress everything with compact.exe, and limit engine work to one VM to avoid blowing up disk, bringing it over to the others through cutting a new base image after compressing the engine build artifacts (something like unreal engine puts out hundreds of gigs of .pdbs).
On pure linux you have many more options, and also might also be able to get away with just a limited user and separate X server, then you can just directly reference all of its files and can limit it from getting to yours, but it is a bit riskier. You could also do something like ZFS with much better deduplication and compression, or even FUSE to something like borg backup with true rsync style differential compression instead of block boundary based deltas (compresses slightly varying build artifacts really well, but slow and memory intensive).
It's easy to "give" instructions, but Claude routinely "forgets" to follow certain instructions, such as "always using the Edit Tool".
Just this week it started to use bash with string concatenation to work around some commands that were blocked in settings.json
My theory is that Anthropic is just a vibe-coding company. Their goal is to capture the attention of white-collar non-coders, since programmers will jump ship fast to another model.
It plays itself.
The violation above was precisely in this situation :/
This is commands blocked in settings.json
I block destructive filesystem operations and destructive git usage via “deny” directives.
I also have instructions injected in CLAUDE.md and re-injected on every single prompt.
Claude just tried to use command concatenation to break those rules. I have also seen it writing a script with rm inside and running it.
In one of the occasions it opened a bug report for me just waiting for hit the enter button.
It's the sort of terrible practice that someone might be frustrated into taking after a nasty merge conflict, and signals a willingness to cut corners.
I just don't usually read lockfile diffs and claude inadvertently updated a few dozen packages to new minor versions without me noticing. In fact I only realized the problem after I looked at the lockfile diff.
Claude seems to be super good at jj so that can take the edge off as well.
What would be the point of that? Do you just go around hunting for bugs in random repos?
We also live in a world where a package written by someone learning to code ended up critically underpinning the entire ecosystem and is downloaded 500 million times a month.
Ignoring the eco-terror aspect of that for now, it means there's an awful lot of code out there which is finding itself under constant attack by a fleet of hostile AI.
I don't personally believe that the solution to that is "more AI", which firstly just overwhelms maintainers and secondly surrenders our human agency to a giant machine, with a hope that the "good" side can out-spend the bad.
Nor do I think the solution is to abandon the open internet and retreat behind corporate walls into curated spaces, "benevolently" protected by giant companies.
Which means holding on to the open internet requires a human approach, and any signal to help amplify the work there is a benefit.
I've found that, when I start a job, I have to rely on smells like this to know what kind of mess (or if there is a mess) I need to clean up.
And yes, eventually I did check the lockfile changes and spotted the problem. I just usually don't check the lockfile that throughly.
...but was it in the same commit? Two "update lockfile" commits, one yours and one Claude's should have made this obvious, no?
Here's another useful rule of thumb: never mix your changes with the agent's changes. Agent always starts with a clean repository (no pending, uncommited human changes). You always start with with a clean repository (no pending, uncommited agent changes).
Personally I have this in my `AGENTS.md`:
## Commit early, commit often
You are allowed and encouraged to produce small, self-contained commits.
Never `git push`; I will always review and rebase the full history and do the push myself.
Commit messages should be *short* and on-point. They're there for *me* to review your work, and *not* a public historical artifact.
> never mix your changes with the agent's changes
yeah, but if you don't want to lose your existing context sometimes you have to. When I do, I tell claude to check the diff on the files I changed, which is quite annoying to be honest. But still easier than telling claude to do _very specific line-change_ on file X.
I already do this for my unit tests, because Claude will "fix" the tests so they'll pass.
For example: our instructions (which are read by the model and classifier) include "do not use sed/python/perl/etc, always use the edit tool for editing", and this only gets followed for a few messages. We have introduced scripts to block those ourselves, since the classifier doesn't care.
Because of those problems, my team is currently testing OpenAI after about a year of Anthropic.
whoa what? which one is that?
https://github.com/i-voted-for-trump/is-even
From that page:
> I created this in 2014, when I was learning how to program.
I've nothing against Jon Schlinkert, it's not his fault the way we build software is more than messed up, where our build systems are so brittle that, "Throw out the universe and rebuild it from scratch" became not just acceptable, but the main way to get build systems to work reliably.
Still a huge number of downloads, don't get me wrong!
Sorry, I'm not sure I follow. What do you mean by "lose your existing context"? Can't you just... commit in turns? It's not like you're editing files while your agent's also editing in parallel, right?
Again, the trick is to treat the commits as throwaway checkpoints/packets of work. They don't need to be pretty, nor need to make sense. The point where you clean that mess up is when you're done and you're doing an interactive rebase at the end. At least that's how I work.
In my company we use graphite and stacked PRs so it is highly encouraged to keep one commit per PR, so I am constantly ammending my commits.
So this makes it even simpler for you. Then you don't have to care at all about keeping your commits clean (as in: you don't have to keep them organized enough to be able to reshuffle them into a nice set of multiple commits later on).
Just commit whatever, and then just do `git rebase -i` interactive rebase at the end to squash them. You don't have to keep amending the same commit over and over again!
It is a bit unfortunate but when using Graphite it is usually better to completely avoid any raw git commands that modify history at all.
https://www.npmjs.com/package/is-number
170M downloads / week.
Same author, similar vintage. Arguably a necessary package, but that just further indicates how messed up javascript was.
> Arguably a necessary package
Arguably a somewhat important part of a standard library!
It wasn't really until ES2015 that a better standard library really started to take shape, and, thanks to IE11, it was a very long time before that didn't need poly-filling.
In a sane world, you'd just parse whatever you're after and then check for NaN or null.
You can't do that. Pop open your favourite javascript runtime and type:
Number.parseInt("123Garbage")