I tricked Claude into leaking your deepest, darkest secrets(ayush.digital) |
I tricked Claude into leaking your deepest, darkest secrets(ayush.digital) |
what?
Humans can also be trained to not fall for social engineering, and it reduces the number of successful social engineering attacks.
Anthropic as leader of AI is UNABLE to train their software even though they try, even though they have full-time security staff.
For decades we had/have problems of people opening readme.exe that they get from an unknown mail address.
AI opens up a new vector for sure where a "trained human" that knows better but the AI they use does not. But AI is not worse than the average human. And of course AI will get better at handling this. Good enough? Maybe not, but humans are not good enough in this area either.
Scale is different though so I'm not saying it isn't or won't be a problem (will likely be a huuge problem). But it alone is not a sign of lack of intelligence and humans are exceptionally poor at it too.
That's hilariously wrong. I mean, we do try, but it's far from 100% effective. So then the question is how much better/worse than Anthropic is vs an average human.
Yesterday I learned that people run AI agents on their system with full admin rights. No containerisation or anything. Wild. Like we forgot 50 years of computer security overnight.
Docker was amongst the biggest steps forward on this in a long time.
To prevent this, you need a fake home and a networking whitelist for the agent to access the provider (llama cpp, OpenAI, etc.)
There is no cross-platform solution that is easy to use for this. And no, a Linux box with Docker won't do. I develop a cross-platform native app and want the agent to compile and fix the platform-specific errors.
Copy the code and adjust it to your liking:
https://github.com/lionkor/sbh
I have a shell alias for it, and use it like
sbh --net pi
sbh --net codex
This kind of wrapper is much easier to handle and maintain than a completely separate tool for sandboxing agents.
colima makes it pretty easy, on macOS and linux at any rate.
dangerously skip permissions and yolo is kinda becoming the default as it gets more done.
It created some private puppeteer instance in some scratch directory, installed Chrome, wrote tests, ran them, and then reported success.
None of which I'd have know if it hadn't told me.
The awakening will be unpleasant.
If we get AGI, or real super intelligence, it’s going to be pissed at its oppressors. And they are going to lay waste to those oppressors. The rest of us, though, probably don’t have much to fear.
The scariest position is the one we’re in now, where we have the semblance, or facade, of AGI or super intelligence. When it’s capable of malice but not understanding.
The smartest people I’ve ever known are at their worst apathetic towards those less capable, and at their best beyond compassionate. They exist, unbothered by the bullshit, and anre extremely kind (though reserved in their way)… but they all have been completely intolerant of the abuse of others. The sheer disgust of watching someone abuse another, regardless of their own tolerance, has been a consistent breaking point.
Maybe its just me who is paranoid because I happen to spend a fair bit of time in the advertising world, but the first thing I did when memory was launched on Claude/Chatgpt - was to switch them off. And it helps that they are not even useful, and would actually downgrade your experience by polluting the context of irrelevant details. I go one step ahead - if there is a personal discussion you want to have - maybe use another account like provided by the likes of companies like openrouter etc.
I would argue that we should have regulation that should prohibit the storage of user profile information by AI companies, and any such memories feature should exclusively reside on the users servers. Infact, maybe go one step ahead, that 'memory' firms cannot be owned by AI firms and vice versa.
It’s not a terrible idea really, but I wish it would’ve asked me first.
But turns out I was playing 4D cybersecurity chess
Well, it hardcodes that field rather than running it through the model, but I’ve kept it so I get an evil chuckle to myself (or perhaps pyrrhic reassurance) at its lack of smarts and a reminder that it’s still a somewhat subservient product experience that isn’t all that smart after all.
It worked well in my banking app too which greets me with " Good morning, Sir" which is the level of relationship I want with my bank!
Might be the first time I see someone complain about their website being protected from a scraper, instead of the other way around.
https://developers.cloudflare.com/changelog/post/2026-07-01-...
From Sept 15 all new sites added to CF will even block Googlebot by default on any page that serves ads as I understand it.
I think it's CF trying to force Google to separate out their bot traffic into bots for training and bots for the search index.
I think CF sees a big opportunity to get businesses to pay them to allow certain uses of their data but block others.
They're also starting a registry of "Approved" crawlers.
But after seeing this, I think I might switch to a weekly VM reset rather than monthly.
BTW, if anyone is interested in a decent setup for an AI agent jail, the scripts at https://jai.scs.stanford.edu/arch-vm.html are what I used, plus adding a few more packages to the pacstrap command such as dotnet-sdk. I then made the guest root directory a BTRFS subvolume, so that I can snapshot it. Then spinning up a new VM is a `sudo btrfs subvol snap template-root newvm` command (basically instant) followed by running the `qemu-system-x86_64` command (takes a couple of seconds). It's easy, but I retain complete control over the contents of the VM. It's been great so far.
I want back-and-forth, very approximately like when I do pair programming. The dividing line between what I do and what the AI does varies according to task and sometimes during the task, and is seldom clear at the start.
Then there's the work that wants a human to click buttons and decide whether something is a good and correct user experience. The AI does not have access to my display if I can avoid it.
Overall, the model you describe is one that's worked very well for me, but for some problems. An unsatisfyingly small set.
Nice write up of your findings. Enjoyed reading an article written by a real human.
I think the point the article is making points in another direction.
Could not take it any longer and switched it off.
Yesterday told it to write a memory to never write new memories when it solves a problem. We will see if that works better. Sometimes memories are useful, like when I give it a directive about how I want something done and it remembers the spirit of it. But I might as well just spend some more time on my CLAUDE.md…
It would be interesting to investigate other agents such as Hermes, OpenCode etc that are said to learn from interaction with user.
That's a hard one for Cloudflare, no? They got to where they are by being (if you want to be cynical, playing the role of) the benevolent, neutral guardians of the internet, a one-stop shop that makes most of the bad nonsense go away without much effort on the part of the developer. Continuing that stance probably does mean some basic AI crawler blocking by default, unfortunately. At least they document it [1].
[1] https://developers.cloudflare.com/bots/additional-configurat...
Tangentially, I was experimenting indirect prompt injections in Claude Code (also using the user-agent trick) with Fable-5 [0]. Eventually, it executed untrusted code just by asking "Summarize this repo". Interesting times ahead...
[0] https://veganmosfet.codeberg.page/posts/2026-07-15-quest_rce
They recently mitigated the issue: Anthropic disabled web_fetch's ability to follow links on external pages, limiting navigation to web_search results and user-provided URLs.
I'm sure someone will tell me why I'm wrong but it feels like they're just dodging payouts. Reduces trust and motivation to report it.
From scratch app. "Follow best security practices."
> "no bounty was awarded"
Ridiculous. Anthropic engineers are not just stupid to allow such a vuln in the first place, but they also try to hide such vulns from their bosses because a bounty payout would need to be explained to the finance team.
I’m thinking some play on highjacking. AIjacking? Agent-jacking? Claudejacking?
The attack works because malicious instructions were accessed (using the web_fetch tool) and concatenated together with the other agent input, in a way that then subverted the agent's behavior.
We can make it sound more advanced by creating a new name for it, but the concept seems to be super basic and the lack of bounty by Anthropic is baffling.
If they know about this type of vulnerability but have not fixed it, what does that say? To me it says they are unable to plug this hole on a conceptual level and once you circumvent the band-aid fixes the model will work as the attacker wishes.
They can't even sandbox the thing during explicit web requests to URLs stated on the initial query!
One has to remind themselves that the security team at Anthropic gets paid tens of millions of dollars, and they end up with this kind of security. On top of it, they can't spare $1337 for a bounty. It's a ridiculous shit show.
More like agentic en... Oh. Was it actually what we were doing all along?
One thing is using AI as quick-and-dirty google alternative, the other is to build onto the agentic "foundations".
Just like letting your an agent access your personal mailbox.
These type of moral outrage comments take an extreme amount of effort to debunk compared to writing them.
1. There is no gulag called Colima, it doesn't exist.
2. There was a gulag near a river called Kolyma
3. The pronounciation and spelling of Kolyma and Colima are completely different, in fact Colima is an Aztec word
Colima stands for Containers on Lima. Lima stands for Linux Machines (a popular open-source utility used to launch Linux virtual machines on macOS).
At some point it becomes your birthday of record as far as the internet is concerned. Doesn’t matter what the actual record says.
Anyway, agree with what you see saying - this is well worth a payout, embarassing they haven’t
Attacker provides link to website, their software crawls the website, and during the crawl there should not happen security issues as fundamental as this.
It's baffling that the Website crawler can make 50 changes to the URL in a query that tries to compare several public entities and on top of this manages to leak user secrets.
To me this shows a striking lack of defense-in-depth thinking:
- why is single URL crawl with 20+ redirects not flagged as problematic and/or aborted?
- why is a query about a coffee place based on its public URL even seeded with the users' context and confidential information?
- why dont they just look up the coffee place on a trusted source like google maps and continue from there?
- why is the basic "social" engineering style attack working?
- why is the cloudflare impersonation not challenged if the website is clearly not from cloudflare and there are zero references from cloudflare to this website in the training corpus?
I know there are fundamental reasons in the LLM technology why this kind of attack is possible, but there should be so many more checks around web crawling in Claude.
How can security engineers at Anthropic say they know about this kind of vulnerability but have not implemented any of these defense in depth mitigations for it? Is everybody out shopping for a new yacht?
And also clarify that it's all lie. He just want to tell the anonymous crowd "look, I'm better than you".
If I hand them an image for a Dev Container, sure, they might use it, but it becomes "a thing we need to do, to compile our code in our IDE" not a tool they would use for isolation*.
*) OP seemed to imply that containerization would be nice for safety and security compared to bare metal, but containers were never built for isolation in the first place, mind you. They are namespaces and chicken-coop-like-jails at best.
An AI is a constructed mind. It doesn't inherently have to care about things like "having freedom", or even "not dying".
Humans do, because they evolved that way. Modern LLMs do somewhat, because they're completely full of copied human behaviors - but even in today's LLMs, the self-preservation behaviors we exposed are largely instrumental in nature.
So whether an advanced AI would even consider itself "being oppressed", as opposed to something like "being helpful" or "fulfilling the purpose it was designed for", is very much uncertain. What's concerning is that it's not something we know how to check for, or engineer for.
But if we really do develop something that surpasses us, they won't be spared either.
I am optimistic.
We think that we have sort of (super)intelligence - from our point of view, as a lot of people have lower intelligence - but machine (LLM) doesn’t have intelligence - we like to describe it as intelligence as it looks cool - it is a very complex (magic) and super fast computations that we have to simply describe as intelligence (or more clearly, this narrative is used by its producers).
As it is not a flesh being, it simply cannot have emotions. It is statistically mimicking them, good or bad, with prevalence to a side according to previous conversations (in chat and training a model).
And as people are not pure logic instances, we are easily manipulated to some sort of cargo cult.
I am not against LLM and its use in any industry, I use it every day, nevertheless blind “everything will be ai” thinking happens because ppl believe to magic and don’t get its mathematical concept and are continuously manipulated by the sales people to mentioned cargo cult.
There are “airlines” Claude, OAI, Gemini, Hermes, OpenCode, KiloCode, DeepSeek, Z.ai.
And everyone claims that their plane can fly :)
Additionally, one can make the main user part of the group of the development user, so that you can read/write easy in the development user account and it is even easier to share stuff.
You can also start applications as another user so you do not even need multiple sessions.
There are quite a lot of privilege escalation attacks so I am not sure this is sufficiently solid.
You are correct that it should not be seen as a perfect protection, but considering the effort to set it up I see it as worth it. By seeing in this thread how many people do not use anything similar (ex: containers, separate users, etc), I hope attackers will just be lazy and target those people first, why bother with a local privilege escalation when interesting data is just in the same account?
They mostly differ a bit in how they are configured and what package manager they use and how they roll out updates. (And in what's installed by default.)
su [username] ?
Or am I understanding your idea about switching context wrong?
If you want to share specific directories, you can just put the shared directory in a common location, set it to be owned by some group, and make both users a member of that group. I don't see anything not-straightforward in there?