Some uncomfortable truths about AI coding agents(standupforme.app) |
Some uncomfortable truths about AI coding agents(standupforme.app) |
> In reality, though, the code review load for software engineers will gradually increase as fewer and fewer of them are expected to supervise an ever-growing number of coding agents, and they will inevitably learn to become complacent over time, out of pure necessity for their sanity. I’m a proponent of code review...but even I often consider it a slog to do my due diligence for a large code review (just because I think it’s important doesn’t mean I think it’s fun). If it’s your full-time job to review a swarm of agents’ work, and experience tells you they are good enough 95%+ of the time, you’re not going to pay as much attention as you should and bad changes will get through.
Another way to look at this is that AI coding agents take the fun out of a software engineer's job. The machine takes many of the fun parts and leaves the human with more of the unenjoyable parts.
Under our new ways of working, you are required to be excited an curious about this evolution three times per day.
Same thing happens here, you get complacent and miss critical failures or problems.
It's also similar in that it "take[s away] many of the fun parts". When I can focus on simply driving it can be engaging and enjoyable - no matter the road or traffic or whatever.
That might be an issue for supervised "self-driving" cars (eg. tesla FSD), but not really applicable to self driving cars as a whole. Waymo seems to be doing just fine for instance.
I find that fun. I work in a 50 year old IT company, with lots of legacy code and technical debt which we have never been able to address - suddenly it’s within reach to really get us to a better place.
Exactly this. I use agents every day to either produce tests for code I've written according to the guidelines I set out for it, or to produce the boilerplate code (which is seldom enjoyable) before I get to add the cool stuff.
Furthermore, when I inevitably get stuck on a thornier section of new code, or revisiting a codebase which I've not investigated for some time, I can use the agent to provide ideas and suggestions of where/how to start/get unstuck.
Like any tool - it's how you apply it to the job in hand (and ensuring the job is relevant) that counts.
That, and never 100% trusting the output.
And throughout this, if I'm ever curious about how the ideas relate to some other topic, I can just ask the agent, "Are we designing XYZ right now? Categorically, is it this?" Lots of really cool discussions to be had.
I might be less enthusiastic if I was just shipping CSS changes and the like.
"I want AI to do my laundry and dishes so that I can do art and writing, not for AI to do my art and writing so that I can do my laundry and dishes."
-- Joanna Maciejewska, author and videogame enthusiast
Puppet, Chef, Ansible they are taking the fun out of system administration, Cloud it's going to take away my job. But what happened is roles changed, SRE, Platform Engineering, dare I say DevOps engineers (whatever those are) all emerged.
Software engineering is going through he same transformation with the same knee jerk reactions from both sides of the argument. Either AI run rampant with no guardrails like OpenClaw people burning 1000s in tokens, OpenClaw calling them randomly at 2am, etc.. then on the other side of the argument we have a petition asking the Node.js Technical Steering Committee to ban AI-assisted code from Node.js core.
AI is here, it's here to stay, I believe that those of us that will be successful will find the middle ground.
If you look at the leveling guidelines for any tech company “codez real gud” will only get you to a mid level ticket taker.
Most companies use some variant of the sprint/"agile" methodology, which means that you as the programmer are similar to an assembly line worker. You don't control the pace, you rarely get the chance to actually finish anything as much as you would like to so you don't get the satisfaction of a finished product, you get little downtime in between tickets or projects to feel a sense of satisfaction at what you have done before you move on to something else.
I totally understand why businesses operate this way. It's simple: if you try not to operate this way, you increase the likelihood that your competitors will release more rapidly and take all your market share. It's the basic evolutionary logic. If you can release a decent but buggy product six months faster than the competitor can release a better and less buggy product, there is a good chance that you will drive them out of business. It all makes sense, but it doesn't result in a pleasant experience for me as a programmer.
The job is also very sedentary and it puts stress on your eyes and your hands. Of course I'm not going to compare myself to a coal miner, but the fact remains that in its own ways the job is more rough on the body than some people might expect. Meanwhile the intellectual and constantly changing nature of the field means that you can never rest on your laurels - the job requires constant mental engagement. It's hard to do it well while multitasking and thinking about other interesting things at the same time.
If jobs in this field did not pay so well, I don't think I'd ever even consider doing this as a career. It just doesn't have nearly enough upsides for me. The only upside I can think of besides the money is that you get to spend time interacting with intelligent people. But one can get that in other places.
Coding with the help of AI is a big improvement for me because just automating away the boilerplate and greatly reducing the time that needs to be spent in doing reading, research and experimentation already takes away some of the main time-sinks of the job. And while I appreciate code's deterministic and logical elegance, I've written more than enough code in my life to not miss writing it directly. I also enjoy writing English, after all. It's refreshing for me to spend my time composing precise English instructions for the AI instead of writing in a programming language. My English skills are above average, so this also helps me to leverage one of my talents that had previously been underutilized. Well, my English skills always came in handy when I needed to talk to coworkers or managers, but I meant that they had been underutilized in the act of actually creating software.
Completely backwards - the fun in the job should be to solve problems and come up with solutions. The fun in the job is not knowing where to place a semicolon.
> Completely backwards - the fun in the job should be to solve problems and come up with solutions.
Aren't the coding agents supposed to be doing that too? You give them the problem, they code up a solution, then the engineer is left with the review it to see if it's good enough.
> The fun in the job is not knowing where to place a semicolon.
That's like such a minor and easy-to-do thing that I'm surprised you're even bringing it up.
man... can we not just accept that individuals have their own motivations and maybe my reasons for wanting to do the job aren't the same as yours?
I think i'm going to let people decide for themselves what they enjoy in their job rather than pretending I know better than they do what they should and should not enjoy.
Who are you to tell anyone what the fun "should" be?
Personally, I find writing code very fun, because building the solution is also very gratifying.
Besides which, in my experience until you actually write the code you haven't proven that you solved anything. It's so easy to think you have solved a problem when you haven't, but you won't figure that out until you actually try to apply your solution
> The fun in the job is not knowing where to place a semicolon.
This can be solved with simple linters, no need for LLMs
If a person needs an LLM to figure where an semicolon goes, a LLM is not going to help them code.
You don't need to be a software engineer to do that.
> Skill Atrophy
Very true, but in the same way moving from assembly -> scripting languages degraded the skill of programmers to manage memory. AI is another tool (albeit one that's on a different level than any other transformation we've had); what matters is whether the programmer understands the intent of the code, not the 0's and 1's that it turns into. That intent is still there, we're just writing it at a much higher level than before.
> Artificially low cost
This heavily depends model to model. IMO OpenAI/Anthropic are likely the outliers here, and I do agree that it's unlikely they'll recoup the training costs for their specific models, but they also legitimized an industry - something that's hard to tangibly price. Many of the models out of China however will almost certainly recoup cost. Qwen 3 had a training price of around $1.6m USD. In the first quarter of this year OpenRouter processed around 5 trillion tokens from it, landing at around 2.5m in revenue (very rough numbers). Assuming a 30% margin, that's already 40% of their training cost in revenue for a quarter from a single platform - their usage in china is almost certainly higher.
The reality is training costs are getting cheaper. I agree that currently the top providers are heavily subsidizing costs, but that doesn't mean you can't drive revenue, they just choose not to as having the "best" model right now gives them clout.
The third point is moot if you run agents in a container and review their code, which you can and should do.
The fourth point is real, but the chance of it actually ending up harming people who use LLMs to write code is IMO remote.
yeah there is only one surefire 100% fix for “prompt injection”: use deterministic solutions ie not machine learning.
----
addendum in case someone tries to make this commonly made point -- i don't use deterministic here to mean "i've pinned the ML model weights after training". i use it in reference to the probability theory stuff of training/models (the boring and complicated maths stuff).
sorry, but this is wrong. the only time “prompt injection” is not a security risk is if no data is ever passed as input to a model, or the model output has no bearing on anything in the world anywhere.
in which case, why bother with the model/system in the first place.
you can exclude this security risk from your threat model. but what you’re saying there is you don’t believe it’s high likelihood that someone would want to or could run an attack, or that doing so would have sufficiently low impact/severity. possibly because you’ve put mitigations in place …
> It's about what you give the LLM access to. You can give it access to your complete DB and APIs, or you can only allow it to operate on a very specific piece of data.
aha! mitigations!
limiting blast radius and/or limiting access to model input. neither of which remove the security risk, but they do reduce the possible impact/severity and/or likelihood.
it’s all about the threat model ;)
Skills do atrophy if you don't practice them, but also, refreshing your memory about some technology you haven't used in a while, or even learning something new, is easier than ever. You can ask the AI questions and try things out yourself very easily. Maybe "just in time" learning isn't good enough, but that's more of a concern based on speculation than a truth.
AI is being subsidized, but also, inference costs are dropping due to algorithmic improvements. For example, TurboQuant [1] looks pretty promising and even if it doesn't pan out, there are plenty of other potential advances like that. Competition might result in AI inference being available at good prices even without subsidies. So, again, more of a concern than a "truth."
Prompt injection: an unsolved mess, but perhaps curated, trusted datasets will be good enough for many projects, so you don't have to expose your agent to the open Internet? It's a similar problem to the supply-chain vulnerabilities that downloadable open source libraries have. A valid concern, but it seems like we'll improve security and muddle through?
Copyright: also an unsolved mess, but kind of similar. Search engines copy the web as part of how they work, but that didn't stop Google from becoming big tech. And sure, Napster was built on copying music and was shut down, but YouTube was also built on widespread copyright violation and it muddled through. It's unclear whether copyright law is load-bearing infrastructure for the software industry or for open source software.
Even if that’s not true, given that hardware and software efficiency gains can be expected to continue it’s likely that this is the most expensive the current level of intelligence will ever be.
The frontier models may increase in price, but only because they’re also more capable. If you hold intelligence constant, price should fall over time.
What if the old timers were actually right? I tend to think they were.
It's absolutely necessary that there's a line of people somewhere who will understand the path from garbage collection to assembly instructions. We can't build upon abstractions only as long as we still run stuff on physical cpus.
But it's also unequivocally true that once we have enough long-bearded oldtimers and newtimers who do understand how writing a Python expression somewhere will end up with a register write elsewhere all the others just don't — have to.
In old times, all you had was hardware and to program you had to understand hardware. But those who then did program and did understand were the few smart people who had access to hardware. Everyone else was left out. Now we have high-level languages, scripting languages, AI, what else. As long as we can maintain the link to hardware by some people, the rest can build on that.
And before them, the electrical and mechanical engineers, without them we wouldn't even have these processors. We all ultimately are dependent on them.
And like that you can go on and on
Currently: human output is copyrighted so companies sign a transfer agreement with employees that anything they produce at work belongs to the employer. The employer now owns the copyright eventhough the employee, depending on jurisdiction, still owns the moral rights (which matter not much more than squat).
With AI: the company uses AI to produce code that isn't copyrighted. The company can take it like any public domain piece of software and incorporate it into their product. Their product is copyrighted by the company. There are even no moral rights that are personal (no need to mention "This product is based on work by Claude").
i mean... it's not the most common, but it's certainly not super rare.
https://hn.algolia.com/?q=i+vibecoded
508 results, yes not all of the hits are actual vibe-coded projects. but then this is just the people who are willing to admit it freely as well. so.
If I can switch to an open source model today, run it myself, and spend 1/10 as much as Opus, and get to about where frontier models were 6 months ago, fear-mongering about how we'll have to weather "orders-of-magnitude price hikes" and arguing that that one shouldn't even bother to learn how to use AI at all seems disconnected from reality. Who cares about the "shady accounting" OpenAI is doing, or that AI labs are "wildly unprofitable"? I can run GLM 5 right now, forever, for cheap.
> in fact, they actually compound the problem by encouraging significantly more usage
because if eliminating training costs makes running the model above cost, the problem is helped by significantly more usage not compounded.
More usage compounds the problem only if inference is unprofitable.
(the article briefly mentions training but that's later).
The other truths that were not about coding agents:
--Skill Atrophy. (Use it or lose it-- another thing we already know)
--The economics of serving code agents at scale (Ungrounded in actual numbers, only OpenAI's miscellaneous statements and annecdotes. Actual cost of running code agents: last gen's mid-tier gaming gpu's will get you reasonably close to Claude Sonnet if you put just a little time in to an agent harness, and its getting cheaper and cheaper for better and better. So, at scale, with real sysadmins doing the hard engineering to eek out every last bit of performance-- well, infra needed for serving these isn't the cost center)
--Copyright. (This passed on the same bad read of a court ruling half the press has been doing for a few years now. TLDR: The Thaler vs. Perlmutter case, which said nothing about output not being protected by copyright. It denied Thaler's attempt to register *the AI* as the owner of the copyright)
> Skill atrophy
If LLMs are so good that you no longer have use for the skill, why do we care about skill atrophy? That skill isn't that useful to most people. There are so many examples of this in human history where it was completely fine and we went on to do higher order things that were more useful.
> Even if they set out fully intending to provide the highest level of scrutiny to all generated code, they will gradually lose the ability to tell a good change from a bad one
If this (first order effect) is actually a problem then it follows that we will naturally exercise our skill of detecting good change from bad ones (second order effect) and the skill will not atrophy? (third order effect). Seems like your "problem" is self correcting?
> At its core, the only defense I’ve got for that response is… this time feels different? Not a particularly rigorous defense, I admit, but I did warn you that this was the squishiest of the issues at hand.
Well, if you knew this perhaps it was better just not to lead with it and spend so many paragraphs on it.
> Some might argue that, even if that time comes eventually, that’s no reason not to make use of the tools that are available right now. But it should come as no surprise that I disagree. Better not to become overly dependent on AI coding agents in the first place so you’ll be better situated to weather the storm (and maybe even thrive) when it comes.
Well this argument didn't turn out to be any less squishy than the first one. It's a self correcting "problem" but you disagree and we should do X because you said so. What was the point of all of this then?
> Prompt Injection
I also think this will likely always be a problem but you can pretty much point at ANY tool we use in software development. Your viewpoint would be similar to saying we should stop using libraries because there's always going to be a vulnerability when you distribute code that somewhere in the chain a bad actor can inject malicious code even if the library was created by a trusted source in the industry. We have plenty of examples of this happening in real life. So far, still squishy.
> Copyright/licensing > I’m not a lawyer! I’m a legal layperson offering my unqualified assessment of some tricky legal questions. Let’s get to it.
Sigh, this entire post is slop isn't it? Bad look for whatever "standup for me is".
edit: Standup for me is something that is made entirely irrelevant by agentic LLMs, no surprise. The irony is rich.
The author wants to be the gatekeeper of skill, quality, and how we develop while they hand feed us slop in the form of their blog posts.
- Skill atrophy is not a realistic concern. Every single technological advancement in human history should have atrophied the skills of people working in those fields, to some kind of detriment. Yet that hasn't happened (in any way any of us would notice). You can still find people skilled in everything that is still produced, and everything still gets produced. If nobody's made it in 100 years and only 1 old man still knows how to make it by hand, sure, that knowledge will die. That isn't true of things still actively produced with advanced technology, like agriculture, textiles, metalworking, woodworking, printing, art, music. If this was a serious concern, we would have freaked out more that COBOL programmers were becoming rare to find to support IRS/bank systems, well before AI existed. AI is not the problem, and rejecting it isn't the solution.
If you're concerned your skills will deteriorate, that's not something to fear. You can always get better at it again if you need to. Your brain isn't falling out of your head.
- The artificially low cost is not something to worry about. We have already invested so many dollars into LLMs that we could ride the current SOTA models for decades and still get way more value out of it than the money we burned to get here. The only reason trillions are getting spent now is insane business people are being insane. You don't have to spend trillions. There are several very small teams of AI labs who produce very good open weight models with tiny budgets. You can make much more money using even local AI than from not using it. The economics are there. Do the math: cost of an AI dev team for 1 year, cost of 1 cluster of GPUs to train on, is an order of magnitude less than some companies pay for their advertising budget.
There absolutely is an AI bubble, and when it pops, so will the US economy (causing a world recession, since the Saudis are also tied up in AI investment), but we will all keep using AI regardless. It just won't be Anthropic or OpenAI we're using. The AI world is already way bigger than that.
- Prompt injections are just one security concern, but they are solveable. The trick is to not use LLMs for everything, like most people are today. You don't allow the control plane to be AI-dictated. You segregate tainted data. You practice defense-in-depth. You lean on deterministic, versioned software more than prompts. With these and more practices, you can do a lot of valuable work safely, even with tainted data.
- I think copyright is dead. Unless there's a global rejection of AI in general (which I don't think will happen), the world will adapt to this new order, where everything is a mix of everything, and nobody really owns anything, IP wise. There is just no way to reject the immense value of AI trained on the world's content. The world will simply change its laws and conventions to fit around AI.
Yes. Now turn off the LLM and make an improvement to that code.
But it doesn't have to be that way. You just have to put an effort into shipping fewer, better features as opposed to more features. The projects I'm working on (e.g. agent orchestration, because who isn't nowadays) have a small surface area and high payoff and thus are uniquely well positioned for this.
If I couldn't use an LLM, I would still work on this, and it would have roughly the same architecture. But because I'm able to go probably 100x as fast, I'm able to be much more ambitious. Or rather, I'm able to discover that my initial ideas were not on point and pivot, and not have any sense of sunk cost
Anyway, to each their own.
Months! That's not a long time.
> while GenAI can improve worker efficiency, it can inhibit critical engagement with work and can potentially lead to long-term overreliance on the tool and diminished skill for independent problem-solving.
https://www.microsoft.com/en-us/research/wp-content/uploads/...
it's a real problem when applied to a population.
Because the LLMs actually aren't that good, so humans are expected to monitor them using the skills they no longer have the opportunity to develop and maintain.
The OP talked about that. Did you miss it?
> If this (first order effect) is actually a problem then it follows that we will naturally exercise our skill of detecting good change (second order effect) from bad ones and the skill will not atrophy? (third order effect).
You're ignoring the anti-human psychological factors: humans are bad at continuously monitoring for occasional errors. The tendency will be to adopt a complacent attitude, default allow. It's not a good environment for developing a skill, compared to actually actively using it.
If humans are expected to monitor them using the skill then obviously they are still practicing the skill and the skill is developed and maintained. Help me understand why it is so difficult for everybody with this opinion to take a another step into their premise?
> humans are bad at continuously monitoring for occasional errors
Let's assume this is true for sake of discussion: That's the job, pre-llm or not. Air traffic control? occasional errors. Software bugs? occasional errors. Department of homeland security? occasional threats
If it's hard and required that we handle the issue, then it's a skill that people will naturally exercise and the skill therefore won't atrophy.
If your argument was true we'd have swarms of people doing accounting by hand instead of using accounting tools because you're worried that the accountants will atrophy their ability to audit the output of the tools.
That's not how it works in the real world and we have plenty of examples of it...
But sure, if your argument simply boils down to "this time...it's different" like the author is arguing, then let's leave it at that. There's no value in discussing it further just like there was no value in the original post. It was just mindless slop to promote "standup for me" which is also something that falls under the category of: "things that are no longer relevant because of llms"
imho “tell them what you want and let them come up with a solution” is a really naive way to use these tools nearly guaranteed to end up with slopware.
the more up front design I’ve given thought to, they are usually very accurate in delivering to the point I dont need to spend very much time reviewing at all. and, this is a step I would have had to do anyway if doing it by hand, so it feels natural, and results in far more correct code more often than I could have on my own, and allows multitasking several projects at once, which would have been impossible before.
Someone posted on here the other day about how they were taking a non-credit writing class in college so as to improve their writing, that was the reason the course existed. 90% of the class was kicked out because they were using LLMs to write for them, when the entire purpose of the class was to improve ones own writing.
Why do you think it will be any different with programming?
How do you learn that without programming?
I'm stealing this.
[1](https://www.techspot.com/news/111233-waymo-admits-autopilot-...) [2] (https://www.bloomberg.com/news/articles/2026-01-23/waymo-pro...)
If you look at waymo's prior blog posts, you'll realize that the people in Philippines aren't making split second decisions that was implied by "you’re not going to pay as much attention as you should".
https://waymo.com/blog/2024/05/fleet-response
>And aren’t they still failing occasionally? [2]
What's the failure rate of humans? For both AI drivers and coders, if they make less mistakes than humans, that's still a win.
5x more than Waymos, last I saw.
No... only if you're charging full boat for that inference. As I said above, loss-leading caps are a in play here. Obviously encouraging people to use more of basically anything that is an all-you-can-eat subscription leads to less profitability. Not sure if we're talking past each other or what.
So it really depends on your reviewing ability how maintainable code you will get. It is a bit of effort to review something “you have done” as thoroughly as something a colleague have done. Somehow I still feel sense of ownership even though the LLM did it.
I like reviewing using GitHub’s interface, so I often do a thorough review in that familiar interface while the PR is still draft, and before I have invited others to review. If I review my own code directly in my editor when the agent is done, my brain isn’t in the right context and can get distracted or skip over something.
If so, I don't care if I wrote it by hand or with an LLM. People who think that building something with an LLM somehow dooms the something to mediocrity are engaging in magical thinking. I can simply use as much or as little LLM as will allow me to meet my quality criteria.
Will the end user care? “Does it make the beer taste better”?
> maintainability is inversely proportional to the amount of time it takes a developer to make a change and the risk that change will break something
https://softwareengineering.stackexchange.com/a/134863
i could be wrong, but i'm pretty sure that end-users get upset when a change takes a long time or it ends up breaking something for them.
just because people are finding that agents or whatever are speeding changes up now doesn't necessarily mean they won't encounter a slow-down later when the codebase becomes an un-maintainable mess. technical debt is always a thing, even with machines doing the work (the agent/machine still has to parse a codebase to make changes).
Right now I have a 5000 line monolithic vibe coded internal website that is at most going to be used by 3 people. It mixes Python, inline CSS and Javasript with the API. I haven’t looked at a line of code. My IAM permissions for the Lambda runtime has limited permissions (meaning the code can’t do anything that the permissions won’t allow it to). I used AWS Cognito for authorization and validated the security of the endpoints and I validated the permissions of the database user.
Neither Claude nor Codex have any issues adding pages, features and API endpoints without breaking changes.
By definition, coding agents are the worse they will be right now.
Non functional requirements: performance, security, reliability, logging etc?
>> The post is factoring in training costs, not just inference.
It is not because training costs are irrelevant here. Training costs do not cause your costs to go up as you accumulate more users.
None of these calculations we're talking about include training costs. You're saying that inference is unprofitable (at least given the subscription plans). I'm simply pointing out that we are talking about inference not training as you stated earlier. You are (very accurately) not talking at all about training costs.
> 5000 line
so that's currently half a developer according to my rule of thumb.
what happens when that gets to 20,000 lines...? that's over the line in my experience for a human who was the person who wrote it. it takes longer to make changes. change that are made increasingly go out in a more and more broken state. etc. etc. more and more tests have to be written for each change to try and stop it going out in a broken state. more work needs to be done for a feature with equal complexity compared to when we started, because now the rest of the codebase is what adds complexity to us making changes. etc. etc. and that gets worse the more we add.
these agent things have a tendency and propensity to add more code, rather than adding the most maintainable code. it's why people have to review and edit the majority of generated code features beyond CRUD webapp functionality (or similar boilerplate). so, given time and more features, 5k --> 10k --> 20k --> ... too much for a single human being if the agent tools are no longer available.
so let's take it to a bit of a hyperbolic conclusion ... what about agents and a 5,000,000 line codebase...? do you think these agents will take the same amount of time to make a change in a codebase of that size versus 5,000 lines? how much more expensive do you think it could get to run the agents at that size? how about increases in error rate when making changes? how many extra tests need to be added for each feature to ensure zero breakage?
do you see my point?
(fyi: the 5 million LoC is a thought experiment to get you to critically think about the problem technical debt related to agents as codebase size increases, i'm not saying your website's code will get that big)
(also, sorry i basically wrote most of this over the 20 minutes or so since i first posted... my adhd is killing me today)
Also documentation in the form of MD files becomes important to explain the why and the methodology.