AI Can Write the Code. Who Verifies It?(mihaisplace.com) |
AI Can Write the Code. Who Verifies It?(mihaisplace.com) |
For instance, if I wrote a web front end that only allowed certain HTML tags and certain CSS properties and no Javascript, I could avoid certain classes of errors entirely.
Or for instance, in C, if you disallow dynamic memory allocation you also avoid entire classes of errors.
These are checks I think I can configure with linters and would give me more confidence that AI generated code did not have certain errors.
There probably are problems like that but you could come up with a set of imports that are "safe" for the AI.
That was never true for software development in other domains I worked in i.e. Fortune 500 enterprise tools, compiler/design tools, networking, and semiconductor.
If the future is agentic, human review will be core bottleneck that needs to be streamlined.
- writes the code
- reviews the code
- designs it
- productizes it
- advertises it
- consumes it
around the world,
AI 'round the world
around the world
(while (true))
The industry is surprisingly slow to adapt and I'm writing two capabilities I'd like standardised in the industry.
I'd like standardisation of Agentic tests (I know, it sounds terrible) but these are tests that are broader and capture fuzzy intent of the system. Unit tests, e2e tests are focused and hard to maintain. An Agentic test (or a behaviour test) is where the agent understands the underlying behaviour intent and creates scenarios on the fly and executes them.
For example, I'd imagine that Claude Code goes through multiple Agentic tests where Claude itself tries to use the software using desktop control. This would capture a much broader range of bugs and its much easier to maintain because you only need the test to have a description of the intent and not all the hardcore plumbing required in e2e tests and unit tests (like mocks ughh).
I'd like a framework where I could simply have something like SCENARIOS.md and the framework runs the scenarios by spinning up the common environment required for the test. The industry is severely lacking in this kind of capability.
The next thing missing is Agentic deployment verification.
So image that you are deploying your service and you'd need to verify multiple metrics are in shape before you promote it to all servers. Traditionally this was done either by manual checks by the developer (tiresome and prone to errors) or just having alerts on a few key metrics (not extensive).
Concretely, I'd like the agent complete access to all my metrics and logs and have it make the judgement to either promote it or not promote it or notify the developer.
You'd argue this is not required, but in my experience, having this capability gives me a lot of sanity. There are greybeards who love to gatekeep their responsibilities and would argue something like "just have good unit tests!!11!!" but that's silly.
Why these are not becoming standard or even required practices is beyond me. I'm guessing that there's a cohort of developers whose mindset is risk aversion and they are not great at intuiting what an LLM is good at and what they are bad at. Unfortunately, a lot of arguments are just silly like "but hallucination", "but just have deterministic tests", "don't overcomplicate things".
Think embezzlement. Think someone with a political agenda who changes the homepage on the way out the door.
Having a single point of control is asking for trouble.
Of course, since AI will be doing the writing and maintaining, there's really no reason for that any more. Let the AI do it all and wait for management to realize they're paying senior salaries to people that are little more than entering prompts, a job that really only requires someone to be able to read and write English.
If you give up on trying to reduce the number of permutations you need to test, you're gonna also (even if not intentionally) give up on testing them.
We knew this, of course. But everything old is new again.
if there are many of them (e.g. unit tests), who generates those guardrails and who verifies their correctness?
if they are a few, then how can they cover vast product capability surface?
Guard-rails are totally ineffective
If I generate a piece of code with LLM, and verify it to be correct; can you trust the LLM output, with the same prompt, in a new session based on my verification of my code, with the same certainty that you’d trust LLVM output for the same source code in a similar environment, ran independently? If no, why?
For instance:
>Traditionally this was done either by manual checks by the developer (tiresome and prone to errors) or just having alerts on a few key metrics (not extensive).
Ignores the places where it's always been important enough to have automated checks on many things.
It sounds like you haven't actually tried to do what your proposing then.
> tries to use the software using desktop control
This is not a solved problem yet. I've tried a number of solutions and none of them were actually good enough for what you're proposing. Don't get me wrong, I'm not saying its an unsolvable problem. Only that it isn't sufficiently solved yet.
At some point the agent rewrites Component X, or X+Y, and now many or all of your tests that hit it directly are invalid and need to be rewritten. That introduces bugs, you're changing the tests and code at the same time, so you don't know if the new ones cover the same stuff as the old one.
I won't be "fully committed to agents" re: test-writing until they reach the point where they demonstrate that they're taking contract and component boundary stability into account in placing their tests. Currently I still need to intervene.
And the tests I point it at doing instead of the ones it does by default are currently catching more would-be or latent bugs than ever before, because we're changing stuff so much faster.
Agents won't make you better, quality-wise, unless you were in a lower-percentile of code-writing ability. But they'll make you a lot faster at raw change count. How good or bad that is depends on (a) the situation the product is in and the tolerance for issues/regressions, (b) how much existing judgment and best practices you can wrangle into position on top of those changes to make the path of least resistance for the agent be smoother.
Often the maintenance of the code is also left to AI, human readability is a bonus.
"Let's break prod and fix it later" is a terrible attitude. It didn't start with LLMs but boy LLMs seem to be making it widespread. Would it kill developers to find problems before deployment?
The breaking bugs we shipped have went up about 40% year-to-year, but the engagement, user counts and revenue has grown much more than before because of the new features that got shipped. So overall its a big net win.
In other words, you are shifting cost onto users doing your QA now.
Most people don't have that luxury, though.
In general it seems like the average user does not really care about these things. Some bugs like partially broken features are live for weeks with growing engagement before someone reports them.
Functionality is secondary to engagement of the user, its much, much more important to serve the right output that will keep the user on the feature than it is to have it working 100%.
I think the fact that defective products don’t make a difference speaks to how deeply unserious much of the software industry has become.
Let's say you're are a "serious" baker, who always puts egg wash on top of his buns. You see another baker open up shop that does not do this. You say that he is unserious and does not do things with "care" while he outsells you, it's you who are unserious and will have to close down shop when you get outcompeted.
In the food prep world, we have Health Departments that come in and shut down unserious businesses like that. In software we don’t.