AI Is Breaking This Thing We Call Trust(terriblesoftware.org) |
AI Is Breaking This Thing We Call Trust(terriblesoftware.org) |
You and I have a baseline. A brain that did most of it’s developmental phase while there was still some unified accepted version of truth (even if dubious)
Future generations will have to develop in a more splintered reality. A world where there is nothing to hold on to. Not even a dubious commonly accepted narrative
This has only ever been true in a relative sense. Until recently, one was expected to understand the software-level code they wrote. But not the machine code, or the computer internals at a deep level. You trust that those layers work. At some point in the past you couldn’t just trust that. At some point in the near future, you will just trust that the software code works. Instead you’ll just debate the spec (the prompt). Your thinking can move one level up the system hierarchy. Everyone’s can. We’ve done it before, we’ll do it again, and this won’t even be the last time it happens. Let go of the anxiety and let it happen.
AI systems are not the same. You can't guarantee deterministic output.
Also, code is still the best language we have to convey our intentions as engineers into function. Natural language is too imprecise and ambiguous.
Understanding the code allows us to understand the intention behind the code and identify bugs, plan and architect better. Inspecting prompts gives you a blinkered view of the system meaning you're more likely to make incorrect assumptions leading to serious bugs.
When I buy a wooden shack from a carpenter, I'd never expect them to core-sample every plank to ensure it's good wood. That's because a good carpenter habitually buys wood from a good supplier, and will have the experience to detect other suspicious indicators.
Similarly, my coworker might not have verified the machine code, but I can be pretty confident they didn't vibe-install a compiler by Ken Thompson that puts backdoors into the binary.
There are always limits.
Engineering human types are often guilty of saying something is low cost but then not doing it themselves. Great, "understand everything".
Trust us trust. You trust an entity. What the author is describing is a new kind of transaction. It's not that AI causes people to lie suddenly (although if you trusted the wrong people this might happen to you, but YOU were wrong then) but now you have a lot of work being done more at the edge of uncertainty and recommendations. Ideas are not ideas anymore, they are working implementations.
So instead of submitting discussions or requests for features, you literally submit implemented features because the cost is lower. This is good. You just have to realize that you are starting the discussion with a working implementation. If you continually try to act like this is not the case, without someone literally lying to you about it, this is more about just accepting a now workflow.
Unlike AI, those layers were traditionally compiled using deterministic tools. And generally came with a suite of other tools to trace and debug them. I don't trust AI in the same way at all.
Just because something wasn't required at all levels or 100% it doesn't mean it wasn't required or it wasn't important though.
I trusted that the machine code worked because it was a mostly mechanical transformation, with little to zero non-determinism involved, and the design decisions for the high level code were still mine and understood. Not the same with an LLM, even when see just as an added level of abstraction.
Given some LLM produced code, I also trust the compiler will produce working machine code matching it. Doesn't mean I trust the LLM code in the same way.
I figure what is happening is (human) software engineering as a discipline, at least on large scale projects, is rapidly and irreversibly mutating to software project management. But instead of managing teams of people, you're managing teams of agents. Where I disagree with your comment is in the assertion that everyone "can move one level up the system hierarchy." Some folks can't, so they'll stagnate. Welcome to the Peter Principle World.
The difference of course being that you had 20+ developers who did understand it. With AI generated code no one has that understanding. The AI sure doesn't
> Some folks can't, so they'll stagnate.
And we have no answer for those people. Just "fuck you the world changed and you're being left behind"? Where exactly does that lead?
If you're developing an agentiv system to build software for you in an automated way using a fleet of agents, you're still an engineer of a technical system even if that system is operating on somewhat fuzzy logic.
Management involves complete delegation and instead the task is to find a capable person for the task and then mostly let them just give feedback so you know whether you hired the right person or whether you need someone else. When you're working with a fleet of agents, you're not assembling a team of heterogeneous workers with different skills, you're hiring the same guy over and over again, hoping that he can do any job and yy same guy I also mean that the guy does not learn anything new over time.
Meaning traditional management skills count for nothing here.
No one made people like Sam Altman into a compulsive liar. No one made Musk call a service full self driving, and then make it return liability to the user the second it decides an accident is about to happen.
Stop removing agency from the people making the top level decisions and reaping all the rewards.
Would you claim that someone who took 5 shots in a row and drove home wasn’t responsible for an accident because the car was what physically did the damage?
I actually have consequences when I make promises and fail them.
I don’t get to keep claiming full self driving will be done next year for a decade and then move on to the Nasdaq changing its rules to get my next company fast tracked into everyone’s 401ks.
I don’t get to commit felony after felony and then do infinite appeals to never pay my fines and move into the White House.
You are making a ridiculous comparison but I guess keep defending the people fucking us all over if you want.
An increasingly large percentage of people being lying POS grifters is breaking trust.
AI just happens to be a tool that the laziest of that group of people leans on a lot to supercharge their grifts.
But our massive and growing trust problem predates and is way deeper than AI.
It'd be along the same lines as saying "I was drunk driving 75km/hr over the limit down a busy suburban road during the middle of the day and 5 people died because of it so we CAN NOT TRUST CARS"
And everyone stands up and claps
I wrote about this frustration a few months ago [1], then realised that if the model providers won't do it perhaps I should try fixing it myself. So, shameless plug klaxon, I wrote an open-source harness that tries to work more like pair programming and less like code review [2].
As usual, my "Show HN" post [3] sank like a stone, so I think perhaps most engineers aren't really interested in this direction. It is possible though, we don't have to just accept whatever tooling the labs throw at us.
[1]: https://philbooth.me/blog/agentic-coding-and-mental-models
My own harness doesn't provide the agent with shell access, so everything has to go through focused tools with limited functionality. There's also a navigator mode where the agent has no writable tools at all, instead it monitors the project for changes you make in your regular editor and comments on them in real-time. That stuff seemed easier to implement from the ground up.
Trust is a fundamental thing, and i cannot fathom how a post-trust society could possibly host life as we know it. Life ad we know it comes from systemic efficiencies that are rooted in trust, requiring it deeply. no trust means extreme loss in system efficiency, in a system already creaking under the straign of supporting current populations.
post trust is in my reakoning a serious issue to engage with.
Who knows whether it was a symptom or a cause. Either way it marks the beginning.
If there was an actual spark, it was the 2016 election as this was the first time when all the disinformation, professional troll farms and Russian influence truly came into play on a world changing scale.
But there was splintering of online communities and the death of nuance for years prior as well. Twitter contributed a lot by giving voices to everyone, and promoting those who’d shout the loudest. Facebook and its nasty algorithm also greatly contributed to all this (thanks, everyone who worked there, for the world we have now… hope you’re proud and putting all that ad-sponsored salary to good use).
Culture war was brewing long before, it just got amplified more and more.
I'll never forget how insane HN went in 2016, felt like basically the whole English language internet went insane. And it's been a decade since then (wow).
And if I can't say that for a coworker, well, that's performance feedback.
While my coworkers are very stable personalities of a consistent work drive, sense of pragmatism, things that they find more and less interesting, career aspirations, etc. they're not all the exact same, and I depend on that as I fit tasks and ownership to people.
Compare a single session. It’ll be extremely consistent.
And high trust societies have spent the past two decades importing tons of H1B style workers from many of the lower trust countries
And based on that, countries like India, Pakistan, Phillipines, Nigeria, Syria all quality as "low to very low trust", and they also represent some of the highest numbers of immigration to many countries right now. A full 30% of immigrants to Canada over the past 5 years are from India, for example.
I don't think it is that unreasonable of me to suggest that people from low trust societies bring low trust behavior with them.
Which does not imply they are untrustworthy people by the way. It does imply a different set of expectations and public behaviors though, which often chafe with their new country
All this, and here we are in 2026. It just shows immaturity and naïveté.