If a module becomes unsustainably complex, I can ask Claude questions about it, have it write tests and scripts that empirically demonstrate the code's behavior, and worse comes to worst, rip out that code entirely and replace it with something better in a fraction of the time it used to take.
That's not to say complexity isn't bad anymore—the paper's findings on diminishing returns on velocity seem well-grounded and plausible. But while the newest (post-Nov. 2025) models often make inadvisable design decisions, they rarely do things that are outright wrong or hallucinated anymore. That makes them much more useful for cleaning up old messes.
In theory experienced humans introduce less bugs. That sounds reasonable and believable, but anyone who's ever been paid to write software knows that finding reliable humans is not an easy task unless you're at a large established company.
Its the same reason a junior + senior engineer is about as fast as a senior + 100 junior engineers. The senior's review time becomes the bottleneck and does not scale.
And even with the latest models and tooling, the quality of the code is below what I expect from a junior. But you sure can get it fast.
I've been doing 10-12 hour days paired with Claude for months. The velocity gains are absolutely real, I am shipping things I would have never attempted solo before AI and shipping them faster then ever. BUT the cognitive cost of reviewing AI output is significantly higher than reviewing human code. It's verbose, plausible-looking, and wrong in ways that require sustained deep attention to catch.
The study found "transient velocity increase" followed by "persistent complexity increase." That matches exactly. The speed feels incredible at first, then the review burden compounds and you're spending more time verifying than you saved generating.
The fix isn't "apply traditional methods" — it's recognizing that AI shifts the bottleneck from production to verification, and that verification under sustained cognitive load degrades in ways nobody's measuring yet. I think I've found some fixes to help me personally with this and for me velocity is still high, but only time will tell if this remains true for long.
Just make sure it hasn't mocked so many things that nothing is actually being tested. Which I've witnessed.
You have to actually care about quality with these power saws or you end up with poorly-fitting cabinets and might even lose a thumb in the process.
It finds spots it missed during the refactor basically every time.
So I partially agree with you, but I think it takes multiple passes and at least enough understanding to challenge the LLM and ask pointed questions.
This is the same pattern I observed with IDEs. Autocomplete and being able to jump to a definition means spaghetti code can be successfully navigated so there's no "natural" barrier to writing spaghetti code.
I was hoping for it to work, but It didn’t for me.
Still trying to figure out how to balance it.
That is completely insufficient for code of any real complexity. All this does is replacing known bugs with unknown bugs.
Citation needed. Until proven otherwise complexity is still public enemy #1. Particularly given that system complexity almost always starts causing most of its problems once a project is further along I don’t think we will know anything meaningful about that statement for at least a year.
Does the study normalize velocity between the groups by adjusting the timeframes so that we could tell if complexity and warnings increased at a greater rate per line of code added in the AI group?
I suspect it would, since I've had to simplify AI generated code on several occasions but right now the study just seems to say that the larger a code base grows the more complex it gets which is obvious.
The conclusion of this paper aligns with the emerging understanding that AI is simply an amplifier of your existing quality assurance processes: Higher discipline results in higher velocity, lower discipline results in lower stability (e.g. https://dora.dev/research/2025/) Having strong feedback and validation loops is more critical than ever.
In this paper, for instance, they collected static analysis warnings using a local SonarQube server, which implies that it was not integrated into the projects they looked at. As such these warnings were not available to the agent. It's highly likely if these warnings were fed back into the agent it would fix them automatically.
Another interesting thing they mention in the conclusion: the metrics we use for humans may not apply to agents. My go-to example for this is code duplication (even though this study finds minimal increase in duplication) -- it may actually be better for agents to rewrite chunks of code from scratch rather than use a dependency whose code is not available forcing it to instead rely on natural language documentation, which may or may not be sufficient or even accurate. What is tech debt for humans may actually be a boon for agents.
So overall seems like the pros and cons of "AI vibe coding" just cancel themselves out.
To me, this sounds like after the transient increase of velocity has died down, you're left with the same development velocity as you had when you started, but a significantly worse code base.
This seems to assume the main cause is the accumulation of defects due to lack of static analysis and testing.
I think a more likely cause is, the code begins to rapidly grow beyond the maintainers' comprehension. I don't think there is a technical solution for that.
Then there's the question of if LoC is a reliable proxy for velocity at all? The common belief amongst developers is that it's not.
-2000 lines of code
https://news.ycombinator.com/item?id=26387179
This is actually one thing I have found LLMs surprisingly useful for.
I give them a code base which has one or two orders of magnitude of bloat, and ask them to strip it away iteratively. What I'm left with usually does the same thing.
At this point the code base becomes small enough to navigate and study. Then I use it for reference and build my own solution.
Yeah, this is the biggest facepalm.
Didn't we grow out of this idiocy 40 years ago? This shit again? Really?
My theory is that at least some of this is solvable with prompting / orchestration - the question is how to measure and improve that metric. i.e. how do we know which of Claude/Codex/Cursor/Whoever is going to produce the best, most maintainable code *in our codebase*? And how do we measure how that changes over time, with model/harness updates?
Traditional software dev would be build, test, refactor, commit.
Even the Clean Coder recommends starting with messy code then tidying it up.
We just need to apply traditional methods to AI assisted coding.
>This yields 806 repositories with adoption dates between January 2024 and March 2025 that are still available on GitHub at the time of data analysis (August 2025).
There were very few people who thought that coding agents worked very well back then. I was not one of them, but I _do_ think they work today.
one more, i swear.
And when people were studying ChatGPT 3.5, everyone would go "Oh, but that wasn't 4!", and when people talk about Opus 4.5 they go "4.6 is so much better!".
My personal position right now is that people are extremely bad at evaluating model output/ changes in model capabilities. Model benchmarks do not reflect the position that models are just 10x better than they were a year ago, but with how people discuss them you'd think that 10x was underselling it.
ofc that doesn't take into account the useful high-level and other advantages of IDEs that might mitigate against slop during review, but overall Cursor was a more natural fit for vibe-coders.
This is said without judgement - I was a cheerleader for Cursor early on until it became uncompetitive in value.
It's basically outsourcing to mediocre programmers - albeit very fast ones with near-infinite patience and little to no ego.
If your ideas/studies/experiences with genAI for software development and engineering were from before januari, basically /clear and re-init.
Attempting to claim the models are the future by perpetually arguing their limitations are because people are using the models wrong or that the argument has been invalidated because the new model fixes it might as well be part of the training data since Claude Opus 3.5.
Also, I'm not blaming users for their shortcomings. I'm just saying they are not perfect but you can get different outcomes according to how you use them.
This is the most common issue I find, even with the latest models. For normal logic it's not too bad, the real risk is when they start duplicating classes or other abstractions, because those tend to proliferate and cause a mess.
I don't know if it's the training or RL or something intrinsic to the attention mechanism, but these models "prefer" generating new code rather than looking around for and integrating reusable code, unless the functionality is significant or they are explicitly prompted otherwise.
I think this is why AGENTS.md files are getting so critical -- by becoming standing instructions, they help override the natural tendencies of the model.
So a cutoff point of August 2025 just before that is a bit unfortunate (I'm sure they'll be newer studies soon).
I don’t question your sincerity. The problem is these arguments and framing are the same ones used by AI hucksters because it’s trivial to move the goalposts. They pull the same tricks as an MLM: success is just around the corner, forever.
IMHO, there’s a chasm between how to actually use these models and what they are being sold as.
The output is certainly prodigious (I can do things for side projects that I'd be very unlikely to finish on my own), but it's not a coding prodigy so it hasn't dislodged me from the idea that it's outsourcing.
Overall, yes, quite remarkable.
On the other hand, I think driving down the need for external dependencies can be a net win. In my experience you usually need a very tiny slice of what a dependency actually offers, and often you settle for making design compromises to fit the dependency into your system, because the cost of writing it yourself is too high. LLMs definitely change that calculus.
I've found agent.md files are more a bandaid then anything. I've seen agents routinely ignore/forget them, and the larger the code base/number of changes they're making the more frequently they forget.
Neither (current) static analysis nor testing is sufficient to score the commit on complexity.
As a trivial example (i.e. probably not something the LLM would do), if the code was a series of 25 if-then-else statements when it could have been a lookup table, no tool is going to flag that.
Now imagine what patterns, which non-junior devs would reject without even thinking, that an LLM would inject. No test, nor any static analysis tool, is going to flag that non-essential complexity.
You can't do this with language models. They don't learn. You can write in your CLAUDE.md stuff like "avoid long if/else chains" of course, but 1) cluttering the context with lots and lots of micro-managing instructions has its own downsides, and 2) you don't want a list of rules, you want deliberate weighing of the trade-offs involved in choices made.
Companies facing this reality are of course typically going to use AI to help manage the increased complexity. But that leads quickly to AI becoming a crutch, without which even basic maintenance could pose an insurmountable challenge.
I would argue they are. Those traditional methods aim at keeping complexity low so that reading code is easier and requires less effort, which accelerates code review.
I've recently given it a go myself, and it certainly doesn't get it right all the time. But I was able to generate AI-assisted code that met my quality standards at roughly the same speed as coding it by hand.
Since then it has been incremental. I would say the big win has been that models degrade more slowly as context grows. This means, especially for heavily vibecoded-from-scratch projects, that you hit the "I don't even know wtf this is anymore" wall way later, maybe never if you're steering things properly.
I think because you can avoid hitting that wall for longer, people see this as radically different. It's debatable whether that's true or not. But in terms of just what the model does, like how it responds to prompts, I genuinely think it is only marginally better. And again, I think benchmarks confirm this, and I quite like Fodor's analysis on benchmarking here[0].
I use these models daily and I try new models out. I think that people over emphasize "model did something different" or "it got it right" when they switch over to a new model as "this is radically better", which I believe is simply a result of cognitive bias / poor measurement.
[0] https://jamesfodor.com/2025/06/22/line-goes-up-large-languag...
I notice on a daily basis even now that it can easily lead to bloat and unnecessary complexity. We will see if it can be fixed by using even stronger models or not.
In my experience, they are not even close.
If you compare it to beautiful code written by true experts, then obviously not, but that kind of code isn't what makes the world go 'round.
I would frame it differently. There are developers successfully shipping product X. Those developer are, on average, as skilled as necessary to work on project X. else they would have moved on or the project would have failed.
Can LLMs produce the same level of quality as project X developers? The only projects I know of where this is true are toy and hobby projects.
The question is, what happens when you have a middling dev iterating with an LLM? And in that case, the drop in quality is probably non-linear---it can get pretty bad, pretty fast.
Read the study to understand what it is measuring and how it was measured. As I understand parent's summary is fine, but you want to understand it first before repeating it to others.
Of course not, you have switched “quality” in this statement to modify the developer instead of their work. Regarding the work, each project, as you agree with me on from your reply, has an average quality for its code. Some developers bring that down on the whole, others bring it up. An LLM would have a place somewhere on that spectrum.