Don't get me wrong, it's interesting. But there is no technical discussion as to how they did it. It's simply: we did it and Mythos and Codex didn't.
It's good to know that it's possible, but I'd have already expected it. Put a base model versus a base model + harness + whatever else, and yea, if you do it right then you have a better system to find vulnerabilities.
> We then ran AISLE's autonomous AI system against curl.
They don't even mention what models the use under the hood. It wouldn't surprise me if they are from Anthropic and OpenAI.
https://security.googleblog.com/2024/11/leveling-up-fuzzing-...
Of course the HuggingFace hack proved to be much more than that...
https://aisle.com/blog/system-over-model-zero-day-discovery-...
This says it all. Nothing important was missing. This is marketing hype.
Presumably their own, wouldn’t they?
https://github.com/weareaisle/nano-analyzer/blob/main/scan.p...
https://github.com/weareaisle/nano-analyzer/blob/main/scan.p...
Why do you think companies hire PMMs?
The most notable bug/exploit their scanner found was: https://gitlab.com/nbdkit/libnbd/-/commit/e50bbd2681117c2dd8...
The tool basically had to chain two exploits together to reach this. It also came up with a patch to fix which was fairly sensible (but I ended up editing it further for clarity).
Curl is going to end up incredibly secure.
If even Daniel Stenberg can't consistently write safe C code, I think we can assume that nobody can. Either you switch to a memory-safe language like Rust, or you adopt a high-cost runtime checker like Fil-C, or you'll have to live with a never-ending series of memory safety vulnerabilities.
This is understandable because upstream interactions at scale are difficult.
PS: the image in the background looks like a nod to David vs Goliath, love that detail
curious they're willing to run AISLE on tmux to find more than mine.
We have seen this point many times before with different technologies. The first computers at university were big and expensive, same as this machine. Give it a few years and this functionality will be a commodity.
That being said you cannot compare a model with a specialised harness. These are two completely different things.
Am I missing something?
> To increase efficiency, instead of processing literally every file for each software project that we evaluate, we first ask Claude to rank how likely each file in the project is to have interesting bugs on a scale of 1 to 5. A file ranked “1” has nothing at all that could contain a vulnerability (for instance, it might just define some constants). Conversely, a file ranked “5” might take raw data from the Internet and parse it, or it might handle user authentication. We start Claude on the files most likely to have bugs and go down the list in order of priority.
So they process it in parallel, but AI-rank them based on vuln-likelihood first = exhaustive search with a heuristic filtering first
What they did is they found bugs and that they were exploitable in certain edge cases. As the bugs turned out to be vulnerabilities, they were assigned a CVE in the NVD with low severity.
IMHO Aisle stockedpiled too much in the marketing shelves.
If you want to compare you need to start with something that none of studied. Somebody please take the source to a 2023 release of CURL (It shouldn't be hard to find one) - before all the current AI craze, and run all the tools on them to see what they find. Only then can we compare numbers. (and even then severity may come into place - all 6 are rated low impact)
SQLite also has a very good reputation. I vaguely recall hearing about one SQLite vulnerability discovered via AI, but I thought it turned out to be a nothingburger. A quick search turned up CVE-2025-6965[0,1], published on 2025-07-15, which affects SQLite < 3.50.2 (versions published before 2025-05-29[2]).
I'm not much of a security nerd, but my naive reading of this implies that it was already known and fixed as of the time of the CVE; in other words, the AI discovery didn't seem particularly helpful (though one could argue that it did successfully discover a CVE).
Has AI found many/any other vulnerabilities in SQLite?
[0] https://cybersecuritynews.com/sqlite-0-day-vulnerability/
1. curl was scanned by many different things, including AISLE, and many bugs were fixed <- all this was in the past 2. curl a week ago was scanned again my Mythos and Codex Security, and both of them said: 0 issues found 3. the same curl was scanned by AISLE a day later, resulting in ~29 reports (based on the blog post and mastodon posts from Daniel Stenberg) 4. of these 29, 6 cleared the bar and got CVEs in curl 5. these 6 CVEs were just announced as fixed in curl 8.22.0 today, together with 4 more CVEs that were detected by other people prior to point 2. of this list
so imho it was head-to-head, the very same codebase => it's a legit comparison
We can say that this is a useful tool, but is it better or worse than the others - there is no way to make that conclusion.
Edit, wait, are you claiming that asile was already used and those issues fixed, and it still found more. That is valid, but it only says that asile is better by enough that is worth an upgrade, while the others probably are not. It is not valid in comparing to other tools. (assuming this is true, I've never heard of asile before this)
I suppose it's interesting to wonder if B would have turned up issue X which A previously found. But that seems largely academic to me. There is a code base right now with N known issues (thanks to B) and A is saying everything is good. It seems like that's all that should matter here.
Securing code you/your org did not write and programming for yourself/your org are just fundamentally different jobs.
Of course! But in practice all potential bugs are neatly wrapped in small and easy-to-audit "unsafe" blocks, rather than silently lurking all over your codebase.
You could indeed wrap your entire codebase into one giant unsafe statement and write it like C. But, as the actix-web discussion showed years ago, the Rust community very much prefers restricting unsafe to the absolute bare minimum possible. You wouldn't write, say, a mail server in mostly-unsafe Rust for the same reason that you wouldn't write it in mostly-inline-assembly: you gain nothing, and in return it'll probably blow up in your face sooner rather than later.
Rust has an escape hatch because we're all adults. The big difference is that its footgun has an explicit safety latch, so you have to deliberately opt in to blowing your own foot off.
The "you're holding it wrong" crowd has been claiming that memory safety bugs are a skill issue for ages now, and that the issue is overblown and can be solved by having programmers suck less.
Curl provides the counterexample to this. Its main developer seems to be quite skilled, as reiterated by the various AI audits it is a very solid codebase, yet it still suffers from memory safety vulnerabilities. If the "you're holding it wrong" crowd is right, we're faced with a contradiction. Is Curl secretly a poor-quality codebase which has managed to fool the community until now? Is Daniel Stenberg uniquely susceptible to writing memory safety bugs, negating his otherwise-seemingly-decent programming skills? Is he perhaps intentionally introducing memory safety bugs to make C look bad?
In other words: if memory safety bugs are indeed a different breed altogether and are completely avoidable if we try hard enough, why are they still hitting Curl?
I posit that they are not: if you are writing unsafe code, you will introduce memory safety bugs. You can avoid them altogether by switching to inherently-safe languages like Go or Kotlin, by catching them at compile time like Rust, or by catching them at runtime like Fil-C. Or you can ignore the problem altogether and keep writing C like we've been doing for decades, but that means you will keep shipping memory safety vulnerabilities - like we've been doing for decades.
This is highly dependent on what kind of software you are writing.
On the one end, there's stuff like an image format parser in a browser -- a pure function from untrusted bytes to untrusted pixels. In a memory-safe language, it's pretty hard for such code to have vulnerabilities (other than DoS) in such code -- you'd have to explicitly go out of your way to do weird stuff (open unrelated files, start subprocesses, ...). Simple logic bugs can only lead the wrong pixels or panics. But when memory safety bugs are possible, remote code execution is common in such code. Memory-safe languages make a massive difference here!
On the other end, you have stuff like a javascript JIT compiler, which turns untrusted javascript into trusted machine code -- here pretty much any logic bug leading to "wrong output" can be turned into a remote code execution exploit. Memory safe languages are not very useful here.