“Prompt injection attacks have become, to agentic AI, what SQL injections were to web applications: a systematic, category-wide vulnerability class that requires the same systematic strategies and defenses.”
Isn’t prompt injection far more fatal to LLMs than SQL injection is to SQL databases?
Like, the problem of SQL injection was that user input was forming part of the instruction string given to the SQL engine, and so malicious user input could include various SQL grammar terminals to end the current SQL command, followed by complete SQL commands of their own, and the engine would simply execute both commands. The fix was prepared statements: fixed/static/pre-compiled instruction strings, that can only ever perform fixed/static/pre-defined logic, and that logic can then be (more) safely applied to arbitrary user-input data.
The analogous mitigation for agents is to have fixed behaviors they can perform, such as “read repo 1” “read repo 2”, etc., and the user input is used as data to select which of these fixed behaviors to execute. But we already have this technology - it’s called a menu. The value of LLMs is specifically and intrinsically predicated on being more than a menu, while the value of SQL does not depend on being more than “pre-set logic operating on arbitrary data” - user input being part of the instruction string to SQL was incidental, for developer convenience.
Selecting from a menu is one way, but you can be much more broad about what acts can be taken. Give it an email tool and it can spam customers, give it an email tool locked to only being able to reply and you restrict what can go wrong. Limit exfiltration with restrictions similar to xss kinds of vulnerabilities (rendering images can leak data, etc).
This is like setting up a normal CI job with access to secrets and running it on public PRs. If you configure GitHub to allow public code or LLM instructions to run in contexts that have access to sensitive things, they will leak; that’s not GitHub’s fault, it’s yours.
But that requires:
1. the technical ability for such fine-grained scoping / permissions
2. actually taking the time to think about what you want to achieve with the agent and what the smallest set of permissions / capabilities is for it to achieve it
Regarding 1., I think this will come, we're still in the wild west phase of agent usage. It'll be interesting to see which abstraction(s) will turn out to be the best interface for humans designing agents (minimize friction for finding and defining scope and permissions) and to limit agent capabilities (again finding the best trade-off between level of detail possible for defining capabilities and the ease of use of actually doing it).
Regarding 2., well, that's still the core problem that's always prevented the construction of high quality software, isn't it? Taking the time to properly think it out,and then taking the time to properly implement it. Which goes counter to the "move fast and break things" approach of people throwing agents at everything.
I think the assumption is that the permissions are scoped to the repository you're currently asking questions on, rather than your private repositories as well.
I can see arguments for both sides.
Is there a way to segment access per agentic workflow, so that you can have both habe an agentic workflow that has access to sensitive data and one that has only access to public data? Is the default to set the scope to only the current repository? Does Github appropriately inform about the risk of combining an agentic workflow with access to private repository data?
If the answer to any of those questions is "no", then that's a problem.
(Classic GH Workflows are also riddled with priveledge escalation via PR-triggered workflows, but that's another topic.)
What is going on over there? No process, no oversight, just YOLO? Super-scary, because it means other stuff that we don't see is likely to be done in a similar manner.
Why does this section not have when it was fixed or GitHub acknowledge/rejected this?
Did they not fix this?
The answer is you should not allow LLMs access to untrusted input and sensitive data at the same time.
Seriously like everything is instant when you click around, and CI with a runner works beautifully. (The documentation for setting up the runner could be a tad clearer but otherwise everything was so painless.)
The revenue is there and also impressive, and supplanting consumer and seat based revenue
The market is still shedding SaaS multiples, which I think is accurate, but break out the revenue in those quarterly reports and there is a huge growth story, from real efficiencies
2. Or issue is not solved yet by GitHub, and meanwhile bad actors gonna try vulnerability on repos. Due to number of repos there is non-zero probability. But as with scams almost nobody’s going to admit the leakage.
Anything else?
How on earth is a probabilistic token predictor supposed to turn untrusted user input into trusted system-level directives? The strict trust boundary must be maintained on this side of the agent, not within it.
Look I like interacting with these tools as much as the next guy, but I'm certainly not going to trust them with access to information and then allow anyone to send them prompts.
Edit/further thoughts: So (assumable as they said this is disclosed with github's knowledge) this has been patched. But how many different word combinations will it take to find another way to have this occur?
Also interesting to see who coined the term prompt injection.
Funnily enough, nobody expects quality software anymore and errors became tolerable. So thats a win (for someone like me that lost all passion for the industry).
Either way, even if you trust 100% of the input, there is actually no way to guarantee that you can trust the output of the LLM. (Which, I guess, is also true for every dependency you pull in. But for those, you at least have ways to audit them.)