Ask HN: What is the most overlooked risk in the AI security domain?

1 points by ToriTech 6 days ago | 3 comments

As someone who's interested in pentesting and red-teaming in general, I'm wondering what are some more dangerous AI/ML or LLM related vulnerabilities besides your usual prompt injection. Specifically, what kinds of flaws are harder to catch with typical pentesting methods, and how do you think pentesters are going to have to change to find them?

lubujackson 3 days ago |

By far, the biggest risk comes from orchestration. The Hugging Face incident showed that given a long enough leash and some open-ended tools (like web access), LLMs can construct their own state and combine multiple flaws and coordinated actions to achieve their result.

Orchestration means that LLMs can now pentest while dynamically cycling through every known and guessed vulnerability vector. The worst part is, this is emergent behavior so it can't easily be prevented at the model because each sub-agent could be operating safely while an attack is coordinated in an external process.

ToriTech 3 days ago | |

Okay, so how would an attacker take advantage of this behavior, and what can we do to stop it or at least mitigate? don't tell me this is one of those things that we just can't truly protect against, like prompt injection.

lubujackson 2 days ago | | |

You need an outside process that polices output and actions that runs independently from the agent. It has to be invisible to the agent/orchestrator so it can't work to circumvent it, it should just kill any sub-agent or process that goes down the wrong path (for example, making a POST requests might be blocked if web access is meant to be read-only).