Some conjecture on my part, but I imagine that the bugs being found are not the kind you can just press a button on the LLM to fix anyway, but are instead the sort of bugs which require the reviewer to deeply consider design tradeoffs, of which I believe there are simply more in a DB or programming language codebase, for example. Stuff I am finding in review is often like "this thing isn't like the other thing, which has this behavior, which is wrong if the aim is X, but sort of correct if the aim is Y, and totally perfect is the goal is Z and the use case is J, but not I. Unless it's Q, in which case you probably need to reconsider some fundamental decision making from 12 months ago and have some conversations with people about how users desire it to work in the first place, and then set about understanding the tradeoffs inherent to that design."

ETA: This is extremely generous to the LLMs, btw. If I got an LLM to pop something this concise out I would fully commit to finding a different profession tomorrow, but for now I am for better or worse stuck doing prompting some non-zero number of times in order to get all of this assembled context, and then need to use my human powers to decide whether or not more humans indeed need to discuss it.

QA is becoming the blocker for projects in an AI era, and the real source of value. AIs can lay down a good base of testing but I see even frontier models still need help with testing because they are basically as bad as humans in terms of testing only what comes to mind, if not even a touch worse. Though they are really good at blasting through problems once they are found.

Ironically, despite the predictions that AI will murder SaaS, I think the result is they end up even more entrenched, because Job Bob's Bait Shack And Tax Filing Services may be happy to instantiate some front office system for themselves via AI prompt, but they aren't going to want to QA that post-AI any more than they want to now. Once we all settle down, SaaS will manage more code than before, and do more QA, and Joe Bob's etc. will be even less interested in trying to replicate the work with AI because it will represent an even larger QA load than it does today, relative to the size of their business.

Or, to put it in the context of this article, even if you can prompt a database into being today, which you can, you can't prompt something into existence that has been tested and QA'd like Postgres, and you would be well advised to ponder that gap between your prompt and billions of computer hours of field testing very carefully before you decide where to store your company's most important data.

Not surprising. I've been running AI reviews through old codebases and finding all sorts of interesting things. AI is significantly raising the quality bar and there is a lot of backfilling to do.

This is one angle of the productivity story. You don't have any sort of plausible deniability covering all these bugs, test cases etc. when they are comprehensively discoverable by tooling. It's like having typing bugs on a strongly typed language with static analysis, you just can't make a lot of excuses.

So that's one measure of extra time that will be spent on every single feature, where in the past things would've been rolled out in a more barebones state. It's better overall, but is it going to be perceived as being more productive?

I remember an LWN article on this question a few weeks ago. The debate was going on and they weren’t sure if it was that there were more bugs or just that because of AI they suddenly knew about more bugs than usual.

Either way, unfortunate but a very understandable decision.

From the article:

"Explaining the number of reverts — features whose rollouts have now been postponed to next year's release, PostgreSQL 20 — Peter Eisentraut, veep at EDB, a PostgreSQL support and consultancy, said the reason could be at least in part attributed to the LLM code reviews used by developers."

Naturally, EDB is an AI shop (https://www.enterprisedb.com):

"Agents need direct, governed access to data at the source. EDB Postgres AI is built for that."

Of course, nothing ever has worked in PostgreSQL before AI!

I guess with LLM as the reviewer, it is going to take a lot longer to be Zero Defect, unless they use LLM as the coder, too.

Seeing how Fable handled Linux kernel bugs, where out of 82 bugs reported only 6-8 were actually bugs (not already fixed or fixed in newest version or straight up made up) I'm skeptical till humans actually sift through this pile.

Using AI to find bugs (rather than write them) is a great use case IMO. Even if you have skilled reviewers, more eyes are almost always better.

Just don’t post the unedited AI “bug report” as an issue please. A human should make sure it’s actually a real problem before dumping it on devs.

This is the happy path for LLM assisted coding. Obviously consumes less tokens being more constrained and focused so it's not what gets the most marketing but having the second loop of code reviews seems worth investment even if the coding loop is entirely manual or you're in full LLM autopilot mode

Training data is filled with shortcuts, conveniences, bugs, hacks, hacks that fix other hacks, hot fixes, patches, mappings that map the hack to hack to a better solution and all types of other surprises.

That's the cost of trying to scale so quickly. You dont filter out the crap.

A nice side effect is that the best way to fix it is more AI.

It's a endless money loop.