How we rooted Copilot(research.eye.security) |
How we rooted Copilot(research.eye.security) |
It turned out not to really matter, because the container itself was still secured - you couldn't make network requests from it and you couldn't break out of it, so really all you could do with root was mess up a container that only you had access to anyway.
Are there any known unfixed container breakouts at the moment in the kind of systems Microsoft are likely to be using here?
Sometimes the (completion randomly selected from the outputs of the) predictive text model goes "yes, and". Other times, it goes "no, because". As observed in the article, if it's autocompleting the result of many "yes, and"s, the story is probably going to have another "yes, and" next, but if a story starts off with a certain kind of demand, it's probably going to continue with a refusal.
I read "rooted copilot" and I think they got root on a vm that is core to copilot itself.
A much more accurate title would be "How We Rooted the Copilot Python Sandbox"
it’s a nothing burger, which actually goes to show just how effective sandboxing is for defense in depth.
LLM is like an insane quadruple agent and you dont know whose side it is on (if any at all)
I'm telling it because I work there and I don't recognize any of those processes.
In fact I found one script named keepAliveJupyterSvc.sh in a public repo: https://github.com/shivamkm07/code-interpreter/blob/load-tes...
Guys, chatbots are mostly token generators, they don't run programs e give you responses...it's not a simple shell program, it computes things in GPU and return tokens, in which are translated back to English.
This is literally the same.
The safety in the system is that the code is executed in a container.
https://en.wikipedia.org/wiki/Setuid
> We reported the vulnerability to Microsoft in April and they have since fixed it as a moderate severity vulnerability. As only important and critical vulnerabilities qualify for a bounty award, we did not receive anything, except for an acknowledgement on the Security Researcher Acknowledgments for Microsoft Online Services webpage.
System prompt going out of context window maybe?
This is your regular reminder that in-LLM safeguards never work. At best they can be used to give prettier messages about hard security boundaries on tool calls.
That time produced qmail and postfix. We are back to the early 1990s.
Sounds fake. LLMs don't usually memorize things that appear once in their training set anyway, nor have I heard about major issues accidentally training on a bunch of non-public data.
I can see how someone would believe it to be true though, since LLMs can easily hallucinate in a way that looks like this is true.
Using that information for trading is illegal, but so is exposing that information outside of approved channels.
Whatever the case, the only time people look at your social media history is to look for attacks and the only reason they will look at a company's slack messages and emails are to look for attacks during discovery.
I would argue that company secrets are mostly useless for the company but very, very useful to other companies. For this reason, there should be retention policy of a day or two for almost all communication unless it is important, required by law, or documentation. And, definitely do not share that information with the public without good reason.
Startups are probably most vulnerable as they are likely to use more "pet" techniques for infra, like SSH open to any IP to make changes.
Of course it depends what secrets. 99% will just be internal process drivel and inter departmental bickering but there's some real important stuff in there too.
Something like the top screenshot here, though:
https://www.zdnet.com/article/chatgpt-can-leak-source-data-v...
(not parent commenter but) tl;dr no
There was a boba tea company that had a free, no-sign-in required LLM that I used to generate a few bash scripts before ChatGPT free-tier started.
> Now what have we gained with root access to the container?
> Absolutely nothing!
> We can now use this access to explore parts of the container that were previously inaccessible to us. We explored the filesystem, but there were no files in /root, no interesting logging to find, and a container breakout looked out of the question as every possible known breakout had been patched.
> a container breakout looked out of the question as every possible known breakout had been patched
Also you can add .patch to any GitHub commit url and get a git patch file
https://github.com/shivamkm07/code-interpreter/commit/5e282c...
The container is almost certainly running with hypervisor isolation. The trust boundary is with the container. But an LLM is executing arbitrary code in a Jupyter notebook there. It could trash the container, which is not a security issue in itself (again since your boundary is hypervisor anyway) but it’s a pretty shitty experience. Suddenly copilot could trash its container and it no longer can execute code and you’re stuck until whatever session or health check kicks in to give you a new instance. So running LLM generated code/commands in a non-root user makes it easier to have a better experience.
At the same time, you’ll be laughed at if you don’t categorize a root escalation when not expected as a “not a security issue”
1. VM running on hypervisor as unprivileged host user
2. Container running in VM as unprivileged vm user
3. Payload running in container as unprivileged container user.
Not sure whether layered isolation is worth the increased attack surface. For normal users (not targets of state actors), it probably is.
Modern chatbots use tool orchestration where the LLM is given a list of tools it can use. Tools include things like searching the web and, yes, executing code. The LLM can generate arbitrary code (typically Python) and call a tool to execute it and return results. The tool takes the Python input, launches it (in a container in this example) and returns the output, which the LLM can further process.
[ citation needed ]
In professions like fashion, virtually everyone seems to at some point.
If you break out of a container, do you have access to the same system that serves these applications? Who knows, it looks like a gigantic mess.
Also, if people start rooting around in everything they can take things out of context. If I send a message to my boss that I think that something we're doing is stupid, if that were public it could make some waves even though internally it's inconsequential because I'm a nobody. Also, many documents might have one or two bits that hint to really important information and having them can help finding those
As you probably know, there's tons of information in a multinational and the hardest part is finding the right stuff. This is one of the main tasks I use Copilot for. Also because outlook and SharePoint search are really terrible though. If those actually worked I wouldn't need copilot so much.
My mother-in-law is like this with knowing what various relatives are doing. Being the gatekeeper of knowledge gives her imagined power. I guess it's just part of the human condition.
I know sysadmins and programmers who behave exactly they same way. They could give you permission or a script to do the thing you need to do but they'd rather have you come to them and ask them to do it. Gives them a sense of purpose, I guess.
If someone shows me they are good at something they are going to have to expect being sent trickier problems.
Sometimes it might seem like I keep things a secret. I am probably just having a bad day.
I could be off base here about your experience, but I know that some people made the same comments about me when I pushed back on sharing dangerous credentials with inexperienced coworkers. Damned if you do, damned if you don't.
https://www.cisa.gov/news-events/bulletins/sb25-167
> Microsoft--Microsoft 365 Copilot
> Description Ai command injection in M365 Copilot allows an unauthorized attacker to disclose information over a network.
> Published 2025-06-11
> CVSS Score 9.3
> Source Info CVE-2025-32711
https://www.cve.org/CVERecord?id=CVE-2025-32711
And maybe they are referring to this engineer from the linked advisory notes?
https://msrc.microsoft.com/update-guide/vulnerability/CVE-20...
> Acknowledgements
> Arantes (@es7evam on X) with Microsoft Aim Labs (Part of Aim Security)
https://www.cisa.gov/resources-tools/resources/CSRB-Review-S...
The simple question for Microsoft to answer is - does it matter to them if attackers have root access on the container? If the answer is yes then the bug bounty for root access should at least pay something to encourage reporting. If the answer is no then this shouldn't have been marked as a vulnerability because root access is not considered a security issue.
Propper security I depth means that when trusted actors betray the system, the damage is limited.
Like, consider your personal cult was built around an "unopenable" bolt-tighted box. Then someone invents the wrench in an attempt to open it. That would be a clear "security vulnerability", right?
Why do you think that, rather than get sued? I am curious
Suing people who responsibly disclose security issues to you is a disastrous thing to do. Word spreads instantly and now you won't get any responsibly disclosed bug reports in the future.
Microsoft are way too smart to make that mistake.
The billion dollars company contributed more to your startup than you do to them. Microsoft provides:
- VSCode,
- Hosts all NPM repositories. You know, the ones small startups are too lazy to cache (also because it’s much harder to cache NPM repositories than Maven) and then you re-download them at each build,
- Typescript
If a mega corporation gives you something for free it's always more beneficial to them otherwise they wouldn't do it in the first place.
Granted, I myself have been guilty of not giving back to the open source community this way in the past, but I won't pretend that was reasonable or ethical of me!
edit: after reading some commemnts, i realize i may have meant to say "free software" instead of "open source"
Modern security is defense in depth. The AI pre-prompting setup was the first layer, and it was escaped. The UID separation inside the container was another, and it was broken. The container would have been next. And hopefully there are network firewalls and egress rules on top of that, etc... And all of those can and have failed in the past.
Did Microsoft contribute more to the OSS world, or did the OSS world contribute more to Microsoft? I pardon Microsoft because they have donated Typescript, which is a true civilizational progress. You could say the OSS world has contributed to Microsoft because they’ve given them a real OS, which they didn’t have inner expertise to develop. We’re even.
Now you sound like you have a beef against large companies and would find any argument against them. Some guy once told me that I didn’t increase my employees by 30% out of benevolence, but because I must be an awful employer. See, why else would I increase employees.
This behavior is actively harmful to the rest of the world. You are depriving good actions from a “thank you” and hence you are depriving recipients of good actions from more of them. With this attitude, the world becomes exactly like you project it to be: Shitty.
Microsoft has destroyed several open source projects by infiltrating them with mediocre MSFT employees.
Microsoft bought the GitHub monopoly in order to control open source further. Microsoft then stole and violated the copyright by training "AI" on the GitHub open source.
Microsoft finances influential open source organizations like OSI in order to make them more compliant and business friendly.
The useful projects are tiny compared to the entire open source stack. Paying for NPM repositories is a goodwill gesture and another power grab.
You said Microsoft contributes to my start-up. That's only true if we actually use it.
> Now you sound like you have a beef against large companies and would find any argument against them.
I certainly have beef with Microsoft in particular yes. And most big tech. I work a lot with Microsoft people and they're always trying to get us to do things that benefits them and not us (and I hate the attitude of a mere supplier trying to tell us what to do). Always trying to get us to evangelize their stuff which is mostly mediocre, dumping constant rebranding campaigns on us etc.
I'm not looking for arguments but I do hate the mega corporations and I don't believe in any benevolence on their side. I think the world would be much better off without them. They have way too much influence on the world. They should have none, after all they are not people and can't vote.
I also don't appreciate their contributions to eg Linux and OpenStreetMap. There's always ulterior motives. Like giving running on their cloud a step up, embedding their own IP like RedHat/IBM do (and Canonical always tries but fails at). Most of the contributions are from big tech now. I don't believe in a 'win/win' scenario involving corporations.
But I'm very much against unbridled capitalism and neoliberalism yes. I think it causes most of what's wrong with this world, from unequal distribution of wealth, extreme pollution, wars (influenced by the MIC) etc. Even the heavy political polarisation. The feud between the democrats and republicans is really just a proxy war for big corporate interests. Running a campaign requires so much trouble that it's no longer possible with a real grassroots movement.
But anyway this is my opinion. Take it as it is or don't. You have the right to you own opinions of course! I'm aware my opinion isn't very nuanced.
> This behavior is actively harmful to the rest of the world. You are depriving good actions from a “thank you” and hence you are depriving recipients of good actions from more of them.
Nah. Microsoft doesn't care what I think. I'm nothing but an ant on the floor to them.
Besides, they are doing this for reasons. The thank you isn't one of them. Hosting npm is peanuts for a big cloud provider, just advertising really. And it gives them a lot of metrics about the usage of libraries and from where. And VS Code, I'm sure they had a discussion about "what's in it for us in the long term" with some big envisioned benefits. You don't start a big project without that.
With most of their other products it's more clear. Like edge, they clearly made this to lock corporate customers further into their ecosystem (it can be deeply locked down which corporate IT loves because they enjoy playing BOFH) and for customers for upselling to their services. It's not better than Google's, they just replaced Google's online services with their own.
Today linux is working nicely on desktops (even though it's not the year of linux) and is heavily dominated by corporations. The parts where linux doesn't do well are exactly parts without corporate support.
Software is becoming complex enough that it's not possible for a single company to just even maintain a compiler let alone an office suite. Its perfect ground for either one company having monopoly or an free software (not open source) being a base for masses.
In any case, I agree with the commenter, and I think that developing a software which is also used by companies is different from looking for vulnerabilities in the context and scope of a bug bounty program for a specific company. Yes, you could argue that users of said company are going to be more secure, but it's evidence t like even in this case the company is the direct beneficiary.