Measuring the sloppiness of code(earendil.com) |
Measuring the sloppiness of code(earendil.com) |
"Please don't pick the most provocative thing in an article or post to complain about in the thread. Find something interesting to respond to instead."
I've taken the provocative bit out of the title above, but please remember that we want reflective comments, not reflexive ones, in HN threads.
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor....
My main feedback for the authors would be, the most important problems for sloppiness are global properties, not local ones. In my experience an agent, like a human, has finite capacity for its attention, but if it runs into local sloppiness that gets in its way, it can fix it on a by-need basis. The technical debt issues that matter are usually global issues that aren't so easy to fix: they require global analysis and global refactoring.
I don't know the answer, but I think we're going to need ways to measure architectural properties, like separation of concerns, clear architectural layering, well-defined interfaces, etc.
If humans increasingly are kept out of coding, then who holds the mental model?
If AI holds the mental model, by definition human prompts will be over lossy channel. This is true without AI too. Software quality is directly dependent on good devs that translate from business/PM speak to technical decisions.
So is coding solved now? it was already solved decades ago.
Imho a better way of thinking about it is in terms of views - you can have an unimaginably complex codebase that nobody reads in full, but either agents or humans receive views/perspectives on that codebase.
Before LLMs we couldn’t have built tools that would deliver such views, now we can.
The agents jot down absolutely everything (and more, which is the issue), thousands and thousands files of perceived business logic, decisions, workflows, architecture, data modeling, etc... It gets stale extremely quickly - we change one thing in the system and 570 "shards" need to be updated. I stumble on more and more things every day that are untrue about the agents "mental model" and it appears to just keep doing it. One small addition not properly reviewed by a human introduces a cascade of misconceptions system wide.
We've tried these "views" you mention, we call them subsections, but it runs into the same problems. One falsification ends up poisoning entire projects.
Remember when we use to refactor functions in a codebase in order to slim down complexity? Yeah - we ended up doing that but with documentation... IMO code is deterministic and we have trained professionals who know how to read it quickly and modify it. (We don't even need to write it anymore, but reading it is the simplest way to ensure no-loss understanding of a piece of functionality.)
Imagine being able to walk into a house and see immediately the leaks dripping from the ceiling, the clunky layout and smell the stinky garbage that hasn't been collected.
We're already in a sort of IDE when you think about it, we're operating a much larger mental model than any code-base, and we rarely remember that we're operating with such a model.
Were you all just full bore shipping a ton with the per token plans? Was it more effective? How many developers @ your company?
It's really interesting to hear that some companies are back-pedaling. We "slowed" down a bit but are still very much using AI and intend to continue using it for... almost everything.
Coding is also a solved problem with unlimited salary budget on the very best developers.
I hate it. My fingers are cramping and my head is hurting.
It's so easy to ship a lot of code, more effort should be put into ensuring the code is correct, with self-improving feedback loops that involve developers, and dedicated tooling...
But again, a while ago, everything was about prompt engineering, and now you can express you idea vaguely and get a somewhat working result, so this likely will evolve fast as well...
Originally I tried to walk the line between improved agent performance and human readability, but current models are so good I don't think human readability matters much, though at a high level, being able to grok the overall folder structure still matters. I've got my hands full polishing a demo for my game, but I intend to revisit Valknut by crafting an eval set that lets me calculate the difference in agent token consumption and task failure rate between ~isomorphic codebase structures. This will let me loop agents to discover organizing policies that improve them.
Truthfully though, with today's models I don't think this sort of codebase optimization is likely to have much impact below 250k-300k LoC projects, and it probably won't be a decisive win till you're near 1M. Also, the shelf life of a product like this isn't infinite as each generation of models pushes those numbers up while also having new policy preferences that require re-evaluating existing policies.
the AI labs are and have been 100% focused on correctness because it is easy to setup and validate.
Adding one more function that almost does the same thing as another will not break anything.
I think this is just a matter of time. At some point there'll be less value to squeeze out of correctness and then the AI labs will start focusing on maintainability. It's probably a lot harder to set up environment to Train for this behavior though.
I tried writing a few skills to encourage agents to spend time thinking about this but it doesn’t seem to generalize very well.
Some benefits can be reaped immediately (cyclomatic complexity), but others emerge over time (churn). A good example of this is say something like a 500 line file that has experienced 2500 lines of churn, and if that rate of churn is trending down or up. Surprisingly useful for understanding you've got a hotspot with an opportunity to pay down debt by spending more time on API design, or just breaking out whatever subset of the code is experiencing thrash.
The funny thing about complexity is that assuming you're lintting/formatting well, you can do a poor mans check by just looking at something like average indentation per line, deepest line, etc.
[1]: https://ouijit.com
I strongly dislike all the "X is solved" narratives. What does it mean for something to be solved? A math problem (or any kind of problem), a riddle, a mystery, a dispute. Those are all instances of a particular situation that requires a "solution", but new situations will always come up. I understand that by "cancer/coding/X is solved" in this new age of (gen)AI that means the ability to streamline or speed up the "solution finding" procedure, but even that presupposes a fixed, static, fully deterministic space of the things we are trying to "solve".
Even cancer cannot be fully solved - Demis Hassabis slowly started drifting away from using the word "diseases can be solved" because they can't be eliminated - we can only speed up the process for finding a cure for any particular disease, be it existing or new and/or evolving.
Is bridge-building solved? Architecture? Why are architects still employed? Is solving "civil engineering" or designing an optimally running machine a thing to be solved? What are we trying to do when we talk about "progress with AI"? Even when the "recursively-self-improving-AIs" and "perfect" robots do arrive, we're still bound to work with them, and they'll have to evolve to find new solutions to new problems.
To be clear, I work with and rely on LLMs every day, from coding custom RAG architectures with CC and Pi to research and agentic data science. These bombastic conversations, however need to quiet down a bit so we can get back to work :)
I does the thing, adds some code, adds test that cover that code, and responds with "Done."
At a fully agentic "No looking at code" company, we now don't have a great understanding of that code, we don't know whether it's a good implementation or not, whether it fits our current patterns, is maintainable, or composable.
We're entirely in the dark, at the behest of an agent at this point - if we want to maintain velocity. (And I would love to know whether that velocity is real or just perceived). It feels like the code part is faster? But also feels like I have to spend a lot more time up front working through a problem to understand it. In the past, I gained that understanding WHILE I was working through the code. Who knows.
I finally gave up and read documentation for 15 minutes and solved the problem.
I will never push AI generated code to production without understanding it, and this is why I only generate small code snippets and copy/paste most of the time.
https://dekobon.github.io/big-code-analysis/metrics.html
https://dekobon.github.io/big-code-analysis/metrics-vcs.html
The only caveat is that it's the same model doing an iteration and then using that iteration as a starting point for the next step. So the model is allowed to write absolutely insane solutions, as long as it can read them back, even if no one else can.
One thing that could be done is to use a separately developed baseline coding model B to evaluate the outcome of each iteration. For model under test X to pass an iteration, not only should it be able to solve starting from the previous solution, but so should B, starting from X's previous solution.
Every IMPORTANT RULE WHICH MUST BE FOLLOWED STRICTLY to not to do something from my AGENTS.md and skills are violated without any hesitation from LLM. Glorified autocomplete doesn't work this way. Positive rules are more stable on this regard.
Code that is overly and unnecessarily difficult and cumbersome for humans to work on.
But given the way humans work on code nowadays doesn't involve actually writing it themselves, there's no such thing as sloppy code anymore. Hooray!
This matches my experience. Before working on an issue, I ask the LLM to estimate net LOCs at the final PR based on the scope. It works well, and review steps do flag inconsistencies. But as the OP mentioned, if you turn this into a hard metric vs "design smell", you can see LLMs code-golfing for oneliners.
I worked for years in Dev Productivity with engineers who had spent their entire careers in that field, and code quality was always the biggest "unquantifiable". Any of the metrics in the literature (cyclomatic complexity, erosion, etc.) quickly became very noisy at scale. Conversely, for any given metric you would find countless bits of code that do NOT exceed any metric thresholds but were clearly low quality.
People have experimented with many things over many years at Big Tech scale, which produces prodigious volumes of code daily. The conventional wisdom was "Don't bother trying to measure code quality."
An interesting observation from an ex-colleague is that probably the best measure of code quality is its comprehensibility, or "understandability". Maintainability, stability and adaptability are natural outcomes of that. But understanding lies entirely in the mind of the beholder! Which is why it's such a subjective metric, not amenable to simple mechanistic measures.
But now, we probably do have a technology that demonstrates some analog of comprehension: LLMs!
Specifically: tokens. Anecdotally and empirically (based on industry reports like DORA and DX etc.) AI coding works much better with "good codebases" (more specifically, strong engineering discipline) than otherwise. I wonder if that can be parlayed into a quantifiable metric like "tokens to grok / LoC" somehow.
So, if to fix something we need to first measure it, and if AI can measure slop, the way to fix slop from AI may be... more AI!
It's better than writing assembly! But it still seems to be in the realm of "a new kind of tool" even for the actual coding part, without even caring about what the code looks like or security or anything like that.
Which is why the best results these days for extracting structured data from PDFs is by having the model do it directly rather than writing code to do it. It literally takes that level of intelligence to be reliable at it.
A common approach is to provide the model with a template or structured schema describing the format you want the data in, and the PDF itself, and it should return a JSON with the appropriate values filled in. It won’t be 100% but probably higher than what you’re seeing now.
Existing tools do text extraction (pdftotext) but not that great when there’s internal numbering and stuff like that. It has to be a heuristic approach, and it won’t be perfect. But it’s an interesting data point that the model isn’t able to figure out the best heuristic itself. Or even make incremental progress towards one after a certain point.
Personally I use only one family for code editing, other families for code navigation.
Models tend to reason about the code using their own intuition and putting other families onto the same codebases may end up with them getting confused. Esp if that other model is as unhinged as Grok.
Also, harness matters a ton. Use a wrong harness with a good model and you’ll get terrible results.
Basing it's findings of LLM as judge on this model, and then proceeding to ignore it. This article can be safely ignored as well.
LLM as judge in harness evals is the way to go, for any of your custom needs. Design the eval well.
I believe that the era of “ai writes tons of slop code” will be a stepping stone in the longer story, and is simply a current gap in the reward functions.
Per the author - if we can get strong measurements of what good code is, we can train against it and close the gap fast. Excited to see more thinking in this area
However it is more likely to be something which can be detached..
Totally agreed that we're not looking at the correct metric right now. Increasingly, code quality will be determined by outcomes.
From this point it take not much more time to add proper ci/cd suite, rework database models and migrate things with now some comfidence.
It works very well on small isolated projects and new builds where resulting tools are produced insanely faster than before and with not comparable average code quality (ai slop is 100x better here) from customer usual habits and expectations.
It's because this approach raise significantly the quality level so quickly and so cheap that I advocate for it.
It however does not work the same way for large legacy projects to revive. For larger projects the reciepe is almost the same but requires significantly more manual testing on each pass and carefull regression check. But it does work.
Customer is happy, they take back "more" control over their legacy horrors at the cost of some extra loc but properly working code. There is no point to fight against "a bit too much code" in my case
I don't want to speak for those that work in elite very sensitive human responsibilities software domains, but in a lot of place this is a revolution in my opinion.
I like how this captures with a metric (lines of code and cyclomatic complexity, some other basically tractable measures) in an automated way, something we all by now intuitively know.
Alan Kay called programming "Pop Culture" some 20 years ago[0]
Something as simple as output length is a hard linear floor for productivity, even putting aside the obvious context problems that you're intuiting, and it's far from being the biggest cost that arises from steering skill. Learning to make a smaller, faster model do the same work with less tokens is a technical domain that a lot of people don't seem capable of learning. I'm not just talking about "context engineering", but learning how to fine tune, post-train, create better harnesses, design inference setups, etc. If we're both using AI, but I'm beating you to market every single time and with a better product, what is your AI usage actually buying you? Yes, competency and skill is this meaningful right now, and it's highly technical. Not the least of which because you know how to describe the problem in way that gives it a smaller solution and requires less iteration.
Most of the labor who understand the technology enough to do those things lives at the companies selling you these services, but you can absolutely learn to do these things yourself right now. It's actually really fun! A hell of a lot more fun than fucking prompting that's for sure.
Where we're at, I would equate it to the early mainframe era where the programmers came with the computer. I'm placing calls that we follow a similar track and the two will end up decoupling, that "model engineers" are going to move in-house. OpenAI will have a ring to it like IBM does today.
AI: "You asked to add feature X. Here are 25 questions that impact feature Z, B, and C in your gigantic codebase"
Developer: I can answer 8 of these questions... Guess I need to go figure out the rest of them.
Writing the code + building the functionality has always been the easy part.
It becomes a very heavy and difficult exercise of it walking you through the implementation and the judgment calls it had to make along the way. VERY exhausting.
There's a silver lining though: you do get to gradually clarify a proto project spec and various requirements, but boy does it take time and energy to re-contextualise when the bot tells you that you should make decisions.
It's an LLM concern too. LLMs seem to do better with well-organized codebases, just like the humans they were trained on.
On the other hand, there are claims that the best languages for robots tend to be terse (allegedly); I tend to regard a rich domain model (static types, not terse) as a major facet of high-quality code.
Subsections - how do they run into the same problem? What falsification? The views I mean are dynamic ad hoc. Every single small task gets its own temporary view.
But perhaps I’m misunderstanding something that you’re saying?
My approach is just ocr-ing with Terra or Gemini flash + checking citations with source both ways. But if I wanted to avoid llm calls, I’d just tell Fable to build a pdf reader directly from pdf binary format. Should be way more robust.
It's just a pretty pedestrian data-munging problem where there's no closed form perfect solution and you have to use various heuristics to get the right result.
As an example, many times it is impossible to determine the order of some words from just position data without considering the meanings of those words. This is why LLMs / VLMs are so much better at this task, because they can look at the document holistically like we can.
Also, funny that you mention patents, something I've worked on in the past as well! If you're looking only at US Patents, the USPTO data resource is much, much better: https://data.uspto.gov/home -- they provide the text in XML format (https://www.uspto.gov/learning-and-resources/xml-resources) which is also pretty complex but wayyyy easier to parse than PDFs!
Do you use a public set of documents? I bet I could almost oneshot this with my harness :p
In other words, how complicated could it be?
Well, if Adobe has been introducing complications and making the format brittle and inflexible over the last 35 years to make it hard for its competitors to write software to process PDF files, quite complicated.
Here’s a public appendix from a recent Federal Circuit case. It has a representative assortment of documents (opinions, briefs, patents, transcripts) but contains only cited pages to the appendix pagination is non-consecutive: https://www.courtlistener.com/docket/68048163/15/ecofactor-i...
The underlying docket is usually 100 to 1,000 times larger than this but it’s similar types of files and the appendix shows the various types of headers and footers that can exist.
Totally agreed. But in this use case, PDFs are the working format, not just an archival format. An offline batch process to ingest the PDFs isn’t feasible. Unless there are some super fast LLMs I’m not aware of that can handle tens of PDF pages per second. It seems like Grok and Claude don’t try to read the PDF directly, they use pdftotext or some Python wrapper over pdfium. But maybe I’m missing something!
Picking specific metrics will probably not work, it would be a mix of Goodhart's law with Bitter lesson. Maybe picking and labeling quality repos, having whole suite of metrics as input features and training some traditional AI classifiers to steer the LLM training.
90% of problems are easy once you know what you actually want well enough for you to be able to ask it from an LLM.
90% of code before LLMs was badly copied from StackOverflow anyway.
That 10% that's remaining, I've see 0, ZERO, nil progress. Windows is still awful. Spotify still doesn't work correctly offline. Youtube search is trash. Jira takes 20 seconds sometimes to load a task. LLMs haven't created a new database or a new game engine or a new renderer or anything like that.
The maths breakthroughs are really more of a testament to the efforts of the last 150 years for maths to be an organised verifiable principle. If LLMs had to practice math they way Euler did, they wouldn't be able to find shit.
(sorry if I sound incoherent, just some thoughts while I'm commuting)
thinking about how to accurately describe the problem at hand and figuring out the simplest way to approach it takes a lot of effort well before you get to the implementation step of coding.
After AI, the volume of that crap and slop has increased exponentially.
This is kinda vague. Correct at what scale? I wonder if there's a measurement on the correctness per scale, and hopefully the scale is not just CLOC.
PR content:
``` Lots of AI slop.... .... .... Note: this will not build due to XYZ .... .... More AI slop .... .... End of PR ```
So the dev hadn't even read the PR comment himself and had blindly posted it!
Code will become throwaway stuff, like the results of AI prompts. Cook it up, test it for adequacy, and run it. When something comes along that adds new requirements, just update the requirements/prompt and make a new one.
Test suites will be important.
TLDR: coding is not solved.
I have 2 projects, one it's a distributed platform, the other one is a general processing engine with an inner workflow engine; Since gpt 5.2 I've tried new models to work in these codebases where the code is of good quality and every time I gave the model a slice of work instead of a single step from that slice the code, the tests, the comments, the docs and everything else has been suboptimal, unmaintainable, complex, bloated and just slop, unless I micro-manage and do many passes.
As a dev when you make a change you consider the broad picture, you consider the user, the codebase, future requirements, maintainability, performance, your team's understanding and some of these you do unconsciously. We are slow but that's for multiple good reasons, you push the organization/understanding forward not just loc of that specific project. I can't count how many PR notes or comments I've added considering teammates or just for a specific team member.
I don't see any way forward for an LLM to reach that unless it reaches general problem solving, my definition of GAI that could tackle software development or "coding" would be a model that doesn't require additional pretraining to solve new tasks or improve how it solves tasks in the future, it would just learn as it's going.
Can everything I mentioned be solved with current generation of LLMs and lot's of markdown and gates? Maybe... but the amount of effort required would be similar to the effort an expert system (pre-llm AI) would require to embed the rules, evolve them, check them everytime... which would require billions or trillions of tokens.
---
off: I really like the discussions around how to prevent slop and bloated code as it's something it would benefit coding even without LLMs and can fit as another piece of automated infra for checking and ensuring code quality, I hope something materializes.
Of course, sloppiness to date can be measured by different shared and interpreted preferences and definitions.
> There are some promising other directions I want to explore, such as coupledness of functions, code churn, cohesion and so on. If you are working on evals and would like to talk, I would be happy to do that: sebastian@earendil.com
Hey Sebastian, I just read your article and it thoroughly resonated with me. I've been working on building something similar to SlopCodeBench, but moreso aimed in the direction of architecture, rather than simple one off "code search functions".
In a nutshell, I'm creating multiple domains of common software architectypes. You can think of these as being as simple as a counter (very common in all architectural explorations worth their weight), todo applications, etc and as complex as an online store, a bank, a wallet, a social communication platform, etc.
Given a single domain, we can extrapolate common functionality that is "higher order" to that domain. Features like data synchronization, functioning offline, sharing information, authentication and authorization, etc all land in this bucket. From a single domain (take the counter, for example), I've laid out my initial plans for the various different levels to concretely observe how bad LLMs are at churn, cyclomatic complexity, poor abstraction planning, etc as follows:
L1: Show a number on the screen L2: Allow a user to click plus or minus and the number responds accordingly L3: Show that number on any device running your software, and keep it in sync with all other devices L4: Ensure that additions and decrements to the count, while a device is offline, will replay to all connected devices once connectivity is reestablished. L5: Introduce the ability to reset the count to zero, and ensure that commutes properly if an offline device triggers a reset or vice versa. L6: Introduce user accounts where I must provide an authorization strategy (login with apple, google, passkey, etc) and now segregate a public counter (which anyone can interact with) from your counter (which only devices authorized with your credentials can interact with. L7: Introduce an action menu whereby I can invoke what is commonly known as a "Command K" menu for the actions that can be performed in the application. L8: Allow me to speak naturally to your application and ask it to "go up", "start over", etc (this exercises your architecture's ability to be "accessible" to agents (and vision disabled folks as well) ... ... ... And so on and so on.
Despite its apparent initial complexity, naive solutions to even the simplest domains will be easy to spot with how many lines were changed vs raw additions (how composable a solution is), that cyclomatic complexity measure you mentioned, how many tokens it took, how many platforms (iOS, android, cli, TUI, react native, react) your application will run on, how long it takes to build, how large the binaries are, how much memory is used during the operation of your software, how semantically similar duplicated code across different platforms etc. From all these different values, we can create a hueristical "architecture score" to benchmark against.
I'm also toying with the idea of enforcing that one must submit a bundle of skills, instructions, scripts, etc that I will exercise with my own harness whereby the submitter has to submit a monetary cost with their submission that a budgeting agent must manage the spend and the agents must yield prior to their submission being scored, lest they forfeit the submission.
I'm just quite tired off all the hype and its exhausting and AFAIK, none of the benchmarks actually produce anything of use. One interesting side effect of aligning incentives in the way I've laid out here is that we will have produced open source, connected software that functions well and solves a whole bunch of business needs that all compose together by definition.
Thanks again for your article, would love to have an e-coffee and chat about if there's potential to collaborate on anything here. Despite how powerful llms are, this is still proving to be a tricky endeavor for me.
Best, Michael
PS: here is a demo of my submission for this first round: https://x.com/technoplato/status/2090902061437030777?s=20
Reality: earandil.com uses 170% CPU in Firefox.
What has this author written before LLMs? Why should we listen to him and his adjudication of "perfect code"?
Cyclomatic complexity is the oldest paper generating grift for college students. There are hundreds of thousands of useless papers about cyclomatic complexity.
Pure cope, at least in most domains. The machines can do it better than the humans.
I think most arguments against that are clinging onto old things that don't really matter any more, like coding style.
If most of us are not coding by hand at all anymore, just directing and iterating with the agents, then coding is solved by a reasonable definition.
Which is more important to you: Velocity to a solution? Or velocity to understanding?
Velocity to solution is default for almost everyone else, especially one-off or low impact / low consequence of failure projects.
Meanwhile, the SWE job market across all companies seems pretty rough right now. Talk to someone looking for a job. Most companies seem to be in a holding pattern - little to no new SWE positions available.
It's a far cry from 2021-2022 but I'm not convinced it's worse for SWEs today than it is for everyone else
Code is an abstract concept that is not bound to the physical world and I imagine that future will have some much more of it that it is difficult to comprehend. Everything will be code and more code will be written than ever before.
Code will never going to be solved. The question is how much humans will be involved and I think the evidence is that perhaps just a bit. However, because we are talking about vast libraries of code even if we are involved in under 1% of all code and decision making that is needed around the code, there are still not enough developers out there to take on the task.
I might be wrong :)
The market is already collapsing in China and India.
So much "bad" enterprise code evolved into that state over years or even decades of small changes. Meanwhile, last year I got to watch an LLM-authored codebase speedrun itself into a similar state in only a couple months. And I would say that the enterprise code was actually better. It at least did its job fairly reliably. The LLM codebase was riddled with defects, so much so that it ate up all our time and our feature delivery rate ground to a halt.
There are two observations that really eat at me:
1. Studies seem to indicate that agentic coding uses 2-10x as many lines of code to accomplish the same task.
2. One of the only really well-established empirical results in software engineering is the strong association between LOC and defect rate.
You can't just rewrite everything. So over many years people are touching small parts of the pie.
If it works it isn't low quality.
then the same applies to AI written code?
I don't necessarily disagree. That said...
Why?
I've been grappling with this myself. There is an easy/obvious answer, but I wonder how stable/permanent it is. If you feel strongly about this, are you willing to unpack your judgement?
its just now the same humans weaponized by AI ship 20x more of low quality code.
Both of those things are very different, and AI shouldn't be the one taking the credit if it's the second case.
I've been doing development, in one way or another, since the 90s. I've worked with dozens of teams from enterprises to startups. Hundreds of developers. The quality of work has been all over the place, but the majority was not great.
I'm arguing that what people today call "AI slop" is already higher quality than what most developers created historically, and the fact that tests and documentation pretty much come for free now means that the floor has been raised.
The quality of AI generated code is not great. Yes, it will get better. It's already better than 65%+ of what regular devs can do AND it is faster to produce, iterate, and release.
When I see AI house style my eyes glaze over. Just this morning I reviewed an RFC from a colleague that he said was a spec for a web service. The document had no introduction, no context, it described endpoints for 2 distinctly different services instead of 1, and made no effort to reconcile why there are 2. It was scattershot with details, some of them important, some completely irrelevant. It was replete with typical LLM-ism.
Basically, it was a dump of a conversation he had with an LLM. As a document to build shared knowledge, it was nearly useless. The only feedback I could provide was a polite "I do not understand what you are trying to build".
But, supposedly, another engineer is already working on implementing this spec. I assume the other engineer just cycled this "spec" into his LLM, and off the two of them went. \o/
They are trying to pull me into their project right now, I stood up some containerization infra for them. But, oh boy, do I not want to join. I looked over their codebase, by LOC the codebase is 35% comments, and a lot of the comments are contradictory, there are dependencies that are not used, there is no tooling of any kind (no type checking, no linting, no PR process), there is no auth (this code is already running in production lol -- they have public endpoints exposed that can be used to scrape/mutate internal company data). Another 30-40% of the codebase is unit tests that test trivial stuff like whether their framework's serializers and ORM work, ex: x=DB.create_x(arg=1), assert(x.arg == 1).
At the intuitive level, I do not understand people who say coding is solved... To me it seems like LLMs are a multiplier (LLMs are amazing, sci-fi level shit), but if you multiply a negative number or 0, you get something that is <=0. Making agentic coding work requires a lot of discipline & expertise.
Testing though I'll mostly give you
2. A lot of it wasn’t. Low quality code/speed serves a purpose for point solutions and scripts etc. That’s not the same thing as writing a core system and if the user doesn’t put any credentials in for an S3 bucket then it falls back to giving information about your own S3 bucket (as I’ve seen just this week).
3. Plenty of companies you can discern the difference between mission critical systems versus “business” systems where if it falls over it’s annoying but not the end of the world.
Blaming workers is always an excuse by poor management.
Hand those people an LLM and they don’t suddenly become competent, but they do start slinging more code.
My experience is a little different. For higher abstraction languages the output is largely acceptable in my work. I always consider that LLMs don't know what I don't tell them and they have limited context to work from. Coding issues I often identify:
* Efficiency. Marginal by default. Coding efficiency problems often appear because LLMs dont usually consider the entire codebase or future plans (although they do guess at some futures). Sometimes they write/name things in ways that are lazy/wasted cycles. Most of the time, they don't.
* Security. Marginal by default. I say they do pretty good. Considering all the failure modes, not so much.
* Maintainability. Marginal by default. Mostly due to the careful consideration of modularity, upgrade paths, etc. while often taking wildly different approaches to solutions without having specific broad instructions. Even then, there can be big gaps in quality.
* Observability. Not acceptable by default. There's usually some consideration and can often one-shot.
* Portability. Not acceptable by default. Good, if you specify what those targets are. Regardless, testing validates this above the coding and models are very good at hitting functional test targets. This is less of an issue in something like Java ofc.
Yes agents can produce code that compiles and runs, but I had to add tools to keep them on track, document their work, follow a process, check their outputs. I also use other AIs to generate developer documentation and review code.
It is like managing a bunch of idiot savant eager-to-please interns, except unlike interns, coding agents do not (yet) learn and improve on their own.
I doubt many people here are brave enough to claim their code does what is supposed to do in every conceivable case. Maybe you have high confidence in the correctness of parts of the code. Correctness of an application is murky though. Things we build are never fully correct, merely correct enough. Like maybe you're responsible for the UI in a web app and you're using your expertise to ensure it gracefully handles display across browsers and a gamut of screen sizes/form factors. But are you also verifying how it works when localized with an RtL script? Are you checking every change you make against CJK?
Its documentation about what the code does not do could fill whole books.
UI copy being full of slop explaining what the software does not do is another problem.
I am not convinced that a lack of negative test cases is an issue.
I do agree it generates too much code most of the time.
From time to time I try to do a pass of coalescing flows and cases and removing dead code to reduce the context and prevent the LLM from tripping over itself. But if it’s exclusively LLM maintained code I don’t care too much if there’s more of it.
This reminds me a bit of a PhD Comics webcomic that confidently claimed we would "never" cure cancer, on the grounds that cancer is not one thing. And I don't know that we will ever actually cure cancer, but that wouldn't be the reason. Correctly noting the problem space is bigger than a layperson would initially appreciate is a lot of things, most of them helpful, but the one thing it's not is a formal a demonstration of optimizing against the problem space as a whole.
The refactor ended up adding 22,000 loc.
I went in there and quickly read through it, laughed my ass off. Reverted the work tree. Micromanaged a new refactor. Net lines of code for something really elegant and easy to reason about was -3k loc in the project.
In case you are wondering why vibe coders are doing 30k loc a day, this is why.
Better at writing code within a huge system, definitely not. Maybe in the future, but as of Astra, Fable 5.1, the answer is still no.
It's just that many (I guess that includes me? :D) assumed that they are better than the actually were.
Really doubt we are near that being solved with non-technical folks + LLMs. I'm seeing people gleefully rebuilding products with the exact same blind spots in their understanding/logic using LLMs. Claude, etc are not seemingly able to "AGI" around goofy asks. The CSS looks a little nicer than their legacy products though, lol.
It's astounding to me that people can see coding get solved and not think every single one of these tasks won't be solved too.
Why do you not think these things aren't going to be completely automated? What makes these tasks special?
Fable and Astra can one-shot video games with compelling novel game loops. They can do systems programming, distributed systems, robotics. I haven't found a weak point.
Seedance 2.5 can make video better than the manual labor of VFX artists, 3D artists, and animators.
Nano Banana and GPT Image can do a better job than graphics designers.
LLMs just solved a Millennium Prize Problem, and there are probably more that will fall in the coming weeks.
Just wait. All of these things will be solved.
There is no "stopping point".
Edit:
Don't anticipate that 2036 will look anything like 2026.
Will Smith spaghetti doesn't stay that way forever. Trillions of dollars will be spent on solving these problems. They will be solved.
If you value humans intrinsically, this is necessarily the loop that will converge. I don't think humans have deep intensional a priori knowledge of the structure of reality. If we did, then we wouldn't need tools like AI because we'd be a superset of that. We can only observe and judge.
If we don't value humans, then sure, I think AI is at the point where it can kill all humans (conditional on sentience and resources etc). Two ways to solve a problem - solve the problem, or eliminate the problem statement. Plenty of easier vectors to eliminate the "problem statement", than say, try to solve problems such as making human life better. If you do value the latter though, there will necessarily be human judgers. That's how it works.
Have you tried one-shotting real distributed systems problems? What was the result and how did you verify correctness?
Tell that to the mountain of failed AI slop games on Steam! As a game dev, building compelling, fun games is not even something humans are good at doing consistently. The AI can build the tech, but it can't make something 'fun' yet (unless your bar for fun is simply that a tool created a thing).
Will all these things get automation? Yeah sure. But the idea that they will be perfect automated solutions applicable in all cases is just marketing, it’s not reality.
We haven't seen that. Maybe when we do, we will start to believe that other things will get solved.
Warning: A Fuckin Liar Was Detected
die
https://www.hays-china.cn/en/press-release/content/%E4%B8%AD...
And the problem isn't just that it says what the software doesn't do, most of the things it claims are in fact meaningless, it's not even clearly describing something the software shouldn't do.
AI I've used isn't a better coder than I am - it's just got a lot more hours in an hour than I do.
Most of the human written code was slop, but the really fundamental and successful stuff we relied upon and which we didnt want to throw away? yeah, not so much. most of that was actually really good.
those EJB monstrosities were routinely swapped out by some saas written in python by somebody who did it properly and werent responsible for a lot of late and over budget projects which barely worked or didnt work.
But even still, cognitive debt is a real, documented phenomenon where we lose our ability to change projects.[1]
We also know coding agents tend to accumulate slop in projects over time without some human intervention[2]
There's also the concern of whether its a good idea to be dependent on an addictive slot machine run by a trillion dollar company to do your work.
We are overly reliant on technology today. I don't see this trend slowing down in the future.
I think that a world where everyone is reliant on technology, and nobody understands how it works is a nightmare.
I see this comparison a lot, and I think it's a trap, because it invites us to confuse scaling duplicates with scaling design changes.
Duplicative mass-production was always core to software from the moment it first became "soft". A factory churning out 10,000 copies of the same book maps to 10,000 downloads of a single software release. The paper and bindings of the book may be below hand-crafted standards, but the words are largely unaffected.
In contrast, LLM-coding is the design and prototyping stage. So if we want to learn from textiles/electronics, we shouldn't be thinking of acres of looms, but instead about fashion-design, custom tailoring, determining patterns for clothes, designing new appliances, choosing circuit layouts, etc.
Just last two weeks I had to slap Fable, three times, to stop writing 1000-2000 lines of defensive code... because of DB columns I just forgot should be NOT NULL. That was it. Nothing else. I told it that, boom, -4800 coding lines: gone.
LLMs defend the status quo and they regularly lose sight of everything bigger than the current PR they are working on.
I too am gradually making peace with the fact that LLM-maintained code does not have to be 100% readable for humans.
But this is not about readability. It's about the data model. So one concession I am willing to make is: don't care too much about the code _BUT_ manually curate the data model. So far: small wins on iteration turns and code volume producing. Too early to tell but for now I am happy with the results.
But lets be fair, if an expert would use AI today to build something with this, I would feel a lot more confident than not doing this.
I would start with the base architecture and add all the guardrails for a distributed system, i might even go so far to leverage the math skills of a frontier model like fable or astra. I would for sure have the proper budget for using Fable/Astra.
If anything, it works more reliably today with the smarter models.
I do sometimes see duplicate functions, which is troubling.
Not too surprised that LLMs also don't "get it" by default?
The speedup of slop production being “worth it” is what we, as a society, are having trouble evaluating at this point in time. In all likelihood it’s worth it only in the short term.
That is never worth it. You're ruining the software you work on when you do this.
What sort of human looks at a group of people and goes "it would be really nice if we made people miserable."
Absolutely disgusting.
I just mean the normal, almost inevitable kind of pressure that comes from the business situation. Management has to somehow figure out strategies to handle the pressure and it has to communicate these strategies and the reasoning for them to the engineers in a constructive way.
Generally, this is the pressure to be competitive and make money. It does the developers no good if they spend so much time writing great code that the company goes out of business.
This is just a hypothetical example, I'm not saying that this is how it would necessarily go in all cases.
"Does that code work?" "yes" "so lets ship it" "but its not good" "but it works right?"
Agents cannot be given a high level goal and then left unsupervised, for hours, without making some dumb decisions.
What people seem to be wanting is for an agent to infer vast complex data from terse simple data, which I think is probably impossible on a philosophical level. There's real information loss in language, and compute can only make guesses at the end of the day. I really don't see how we bridge that gap.
I was very optimistic about it when it was announced and saw all the demos, but a week later I find it only marginally better (and in some cases worse) than before.
These companies pay management more than workers for a reason, if you can't even admit that they are to blame then what are you trying to do here? Just attack workers for what reason exactly? Being anti-worker is a great tell to never trust a person.
Who even is supposed to be training us? We're supposed to be the experts. Unless you mean mentorship, which is also generally already a thing at any company that has more than a handful of engineers.
No one is training us because there are no regulations in our industry. Sorry but I still reject the premise of the other poster.
Also yes, in other industries professionals do complain when they aren't given their mandated time to learn while working on the job.
Training doesn’t solve every problem, the worst programmer I ever worked with that a PHD in computer science. Everything he made was horribly slow, wildlife overly complicated, and buggy. Worse he wouldn’t listen to anyone correcting his issues. He’d store numbers in the database as strings to be database agnostic etc.
You do not like bad workers, management doesn't care. They pay bad workers the same as you, bad workers can get promotions the same as you, you will also get laid off with the bad workers as well; or maybe even worse, the bad workers get promoted into management themselves. How do you want things to change in such an environment?
You have no authority to do anything meaningful as a single worker, what if you were given a voice to actually make these claims and have other workers decide what to do based on your voice?
Workplace democracy seems like an interesting concept to explore if you truly want to create better environments with beneficial outcomes to all, not just the few:
https://en.wikipedia.org/wiki/Workplace_democracy
Or are you just trying to bash your co-workers so you become the next pick me?
Only poor leaders ignore their workers, which is what you seemed to have actually experience.
They're generally not going to teach you how to program or what the latest technologies are or what the best ways to do things are because that's literally what they're paying you for. But IME they definitely give you plenty of time and autonomy to be learning things at work.
No stock owners are ultimately in control of public companies. That doesn’t fit any of the models you just described.
Similarly companies are not independent of government control which inherently separates them from management systems associated with governments. A CEO is limited by the law in ways that a dictator isn’t.