I think the framing started in the right path and then took a slightly wrong turn.
Both loops presented benefit from being tighter, faster. One to take a system to a “stable” (maintainable) setpoint quickly. The other to handle uncertainty.
And the additional insight about splitting the systems to better adapt to AI… we’ve described spikes for years, well before AI went mainstream
Many vendors seem to be learning (or not learning, but just throwing their weight against it anyway) that adding hastily-generated AI features are causing customer dissatisfaction, as more people brand the features "slop".
In the best case, the users give the company more chances. Infinitely more chances.
In a worse case, the users assume the new feature will always be bad, given their first impression. It's hard for a vendor to make people reconsider a first impression.
The absolute worst case is that AI enables a new market, but the first attempts are so poor that the first movers make people write that market off as a dead end, leading to a lost opportunity.
Every codebase includes parts that are more experimental, and parts that are more core. My sense is that AI can help on both of these fronts (I.e building rapid prototypes on the fringes and hardening the core with better test coverage).
There’s a speed limit, because the faster you go the less room for error you have. It’s the same as being heavily leveraged with debt. If you have a cash investment and it drops by 50% you can just wait. If you’re leveraged 100-to-1, a 1% drop forced liquidation and wipes you out.
I feel like this is shooting from the hip from a single point of view from some semi-large corpo.
I saw this yesterday
https://trinkle23897.github.io/learning-beyond-gradients/
They are very remotely related yet somehow very close.
Want me to communicate my expertise? Give me some time to actually do it.
This is just an assumption and the whole article falls flat if this turns out to be wrong. In my limited (as everyone else's) experience, working with agentic AI needs good documentation, good specification (spec driven, you know it's all the hype nowadays). Those alone lead to much improvement. Now take into account that probably your senior dev also has more time to think about the big picture, to improve all those little things that were a nuisance in the past but now are a mere "Claude, fix that" in a worktree away.. I would not bet on the assumption of this article.
This is why part of a senior developer’s job is designing and developing the fast version in a way that, if it goes into production, won’t burn the building down. This is the subtle art of development: recognizing where the line is for “good enough” to ship fast without jeopardizing the long-term health of the company. This is also the part that AI is absolutely atrocious at - vibe code is fast, that’s the pitch, but it’s also basically disposable (or it’s not fast - I see all you “exhaustive spec/comprehensive tests/continuous iteration” types, and I see your timelines, too). If you can convince the org that’s the tradeoff, great, but I had a hell of a time doing it back when code was moving at human speed, and now you just strapped rockets onto the shitty part of the system and are trying to convince leadership that rocket-speed is too fast.
I was given a chance to redesign and it and when I failed to add the added complexity I was let go.
To this day I reckon the higher ups are still having the same age old problems and excuses from their underlings regarding a system that has an utterly useless design. The guy in charge, rarely in the office, calmly explaining its a fantastic implementation, the new coders we are getting just cant work with it / operate it well because they suck.
I am not bitter, if anything it just made me terrified of being C-suite of any large company, knowing it would be almost impossible to understand why your company is failing.
I agree that the punchy staccato and the rhetorical questions smell AI-ish, but the way this person uses them, there’s, like, a payload each time. Versus LLM-speak, where the assertions are at best banal and more frequently just confusing.
There will be different shades of usage and maybe we draw a line somewhere in there.
So even if AI was not used to write an article, it could "smell" like AI to someone who consumes less of it.
are we just trying to say, "use AI for prototyping and customer demos that aren't important to be mature, use senior devs to develop and maintain the real products" ? You could just say that then...? Which I also disagree with as how AI should be used, AI is valid to include as a tool across all forms of development - it just should never be put in charge for production-level software (e.g. no vibe coding of mission critical components).
If only higher-ups would recognize that. Instead we see left and right mass layoffs, restructurings and clueless higher-ups who clearly drank not just a bottle of koolaid but a barrel.
> The ‘Speed’ version allows the rest of the business to continue learning from the market, as the senior developers build a trailing version of the system that’s well-reviewed and understandable.
Yeah... that doesn't fly. The beancounters don't care. The "speed" version works, so why even invest a single cent into the "scale" version? That's all potential profit that can be distributed to shareholders. And when it (inevitably) all crashes down, the higher ups all have long since cashed out, leaving the remaining shareholders as bagholders, the employees without employment and society to pick up the tab. Yet again.
No-one says this.
Fine, then, I'll keep the experience to myself.
I love this sentence.
The senior should also start using AI to increase the amount of work done to stabilise the system, in a careful manner. More benchmarks, better testing, better safety net when delivering software, automated security reviews, better instrumentation, and so on.
> And this is how AI affects the two loops
There should be another image illustrating that the amount of mitigations done from senior side, red-/blue-team style.
There's a place for prototyping and experimental features but now agile has cultivated extreme learned helplessness and everything is an A/B test because there's no longer any ability to judge whether something is good or bad based on a holistic vision.
And push an insurmountable pile of technical debt onto the successor.
Well, yeah, I understand the idea and I'm all for it: the less code the better, the less changes the better.
However in certain industries it is no longer a right approach for the job. In modern frontend development if you did not update your codebase for like a couple of months, this codebase falls so much behind that it becomes way more expensive to push an upgrade as compared to daily minor updates of packages. Yeah, I hate this as much as you do, but this is the pace frontend is moving at, and if you don't follow, you will mount technical debt.
complexity bad
say again:
complexity very bad
you say now:
complexity very, very bad
Bro & I would not get along well =)))) But the article IS good stuff.
The message that hits for me is that of AI being a destabilizer while simultaneously being an accelerator. The Speed/Scale suggestion won't address this. A codebase no one understands, growing at machine speed won't go away just because you drew a box around it. The fix is likely more mundane stuff like process and role shifts, smaller PRs, tests, tooling, ownership principles.
“I found this new tool and it’s pretty cool ...”
“This company <company totally unlike the one we’re in> does things this way, so …”
“Here, look at this HackerNews post that says this is best practice, we should probably …”
The safest answer an engineer can give is "no".
The "speed" loop reminds me a lot of RAD. In fact, AI might be _the_ thing that helps us deliver on RAD's promises from decades ago.
https://www.geeksforgeeks.org/software-engineering/software-...
I guess the author has never worked on a dog shit system with no tests at all and constant downtime.
I have worked with “complexity averse” engineers who would rather fix the edges over and over again, than roll up their sleeves and just get the job done.
I just don’t believe that using new tools is at odds with avoiding complexity.
Sometimes you have to take it to the chin, and get to use the new shiny thing along the way to move much faster.
That is to say, the rolling up of sleeves usually involves first adding tests for existing behavior and then doing more-or-less what’s described. The tools involved are aside to this; “catch as catch can”
An AI agent using a web browser like a human. I used various stealth technologies to achieve this. I set it off on a research task for me and it saved me $30 of a purchase by finding the best price. Its Jeff Bezos worst nightmare, visting amazon.com and ignoring all the product placement ads.
It had multiple tabs open, did searches in multiple places, opening products and checking sites....it looked just like a human would do doing the same task.
This I can assure you would not have been possible without my expertise. I had to be very careful to remove all bot signals from the browser, including going to browserscan.net to check. Once done, most captchas were never shown to the agent. There is a NodeJS codebase involved that I wrote by hand.
I searched through the code of the browser automation framework I was using, looking for ways to make it look more human. I had AI help with this part, but had to confirm everything and pull the agent up when it suggested bad ideas.
Most of the work was architectural, including making sure my browser was easy for the agent to use.
I'm going to add 2captcha as a next step, to solve the few captchas that it still encounters (as I still do sometimes as a human).
I'm thinking of open sourcing it, but i'm not sure if its a good idea as if it became widespread, it might encourage the adoption of even more invasive anti-bot measures.
Remember that the first half of this statement, the part listed here, is great. I love playing with new tools.
The only bad part is the implicit bit after the dots: "we should use this in our product." You don't want cool things anywhere near your product, unless the cool thing is that they remove complexity.
Almost all business presidents, CEOs, and owners are thinking this. I guarantee you they are sick and tired of developers taking forever on every project. Now they can create the apps themselves.
My comment isn't meant to debate every nitty-gritty detail about code quality, security, stability, thinking of every aspect of how the code works, does it scale, etc. All of those things are extremely important. However, most leadership never cared about any of that anyways. They only heard those as excuses why developers took so long. Over the last decade they put up with it begrudgingly.
You know all the developers that wanted to complain about IT, cybersecurity, DevOPs, cloud architects for getting in their way and if they only had administrator access then they could get everything done themselves because they are experts in networking and everything else? Well, those developers are about to have the worst day ever when every single person on the planet can generate code and will be "experts" in everything as well.
And society is beginning to suffer from it. AWS alone managed to slop itself into outages twice in a matter of a year [1] (and I bet that's just the stuff that escalates into mass-visible outages, not the "oh, can't start a new EC2 instance of a specific type for a few hours" kind), and a lot of companies were affected.
It's always the same game: by the time the consequences of the beancounters' actions come home to roost, they have long since departed with nice bonus packages, leaving the rest to dig out the mess.
[1] https://www.theguardian.com/technology/2026/feb/20/amazon-cl...
It’s not difficult for seasoned engineers with deep technical backgrounds to whiteboard a distributed system in twenty minutes. It takes hundreds of customer discussions, invalid hypotheses, and years of experience building judgment about whether this is the right solution at the right time.
The engineers who compound quickly have usually built their skills in both areas concurrently. Communication of the latter is more challenging due to the judgment-based foundation beneath it.
That includes gate-keeping behaviour such as not handing off knowledge, sham performance reviews to prevent ambitious juniors from over-taking them (even with AI) and being over-critical to others but absent and contrarian when the same is done to them.
That leverage does not work anymore in the age of AI as having "expensive" seniors begging for a pay-rise can cost the company an extra amount of $$$. So it is temping to lay them off for another one that is a yes person that will accept less.
In the age of AI, I would now expect such experience to include both building and working at a startup instead of being difficult to work with for the sake of a performance review.
Honest question does high velocity / first mover ever really pay off these days?
I don't feel like having the first AI slop to the market has actually paid off for anyone? Am I wrong? Am I missing something? Am I out of touch?
The way I see it, first movers do a lot of work proving the idea works, and everyone else swoops in with better product or at least at a cheaper rate.
Beyond that, let's take the company I work for, for example. We have an ingrained and actually relatively happy customer base on a subscription model. I feel like the only thing increased velocity can do is rapidly ruin their experience.
Maps to what we believe on our team - functional vs non-functional. AI ships functional features fast but developers are more important than ever in making sure the non-functional aspects are taken care of
>> “AI agents are the future of software development. We won’t need developers anymore to slow down the progress of a business.”
> And so, to me, a copywriter, what’s happening here is that the same message is meaning two different things to two different audiences.
I couldn't tell whether to parse this as "We will be faster without those slow developers", or more cynically as "We don't need developers to slow us down; We can now be slow with ai agents". I suspect that with creeping complexity the latter reading will hold up better for large projects.
I am really skeptical of arguments based around "I can do things the model can't" because that space of things is not very large and is getting smaller every day.
The opportunity to not merely cling on to what we have another year but to grow is to say "together, the model can manage so much more complexity than before that we can do things that were not previously possible."
We haven't identified too many of those things yet, but I am certain they are coming.
Literally what people thought after Fan Hui (2-dan) was beaten. For humans software requires ingenuity and creativity. Computers can cheat that, infact computers ALWAYS cheat that to beat humans. NTP as a method of cheating is slightly more general than say board evaluation, so it's less efficient for the same problem, but scaling laws show that with enough compute NTP can beat humans at chess (or any most other arbitrary games, in real time).
The thing breaks, the salesperson says "can you check this out?" then disappears and we're back to where we started.
I don't even find this very new: many companies I've been at have tried to spin-off a "fast" team to sell stuff.
But reduction is narrower than management which is narrower than organization.
Also uncertainty is part of complexity. Being able to isolate what is deemed predictable under clearly identified premises is the best that can hoped on that matter. It means that then one strategy can be applied to protect the stable core, and other strategy can be tried on what is unknown (known and unknown unknowns).
But senior devs are also expected to have a compounding effect even pre-AI. Writing a single doc, refactoring legacy code to make it extensible, building security frameworks specific to the project and many more. All of these would compound the dev team.
I think the same will happen with agents working on a org specific paved path set by senior devs.
I have personally noticed this a lot how multiple people can work on the same problem, but the more senior developers get way more miledge out of AI compared to those that are early in their carreers.
Another difference I've noticed is how many agents one can keep running without losing awareness.
It generally just raised the bar on what management will expect from developers which will result in a shrinking workforce. The only ones that will benefit are AI companies and the upper management since less employees means less management so lower management will get screwed too.
Jevons paradox is already rearing its head, I've seen data suggesting open roles in tech are at their highest since the post-pandemic slump [1]. If you're a senior leader at a company and your engineers are now capable of multiple-times more productivity, is the logical choice to fire half, or set way more ambitious goals? One assumes engineers are hired because their outputs are worth more than their cost. If outputs, at least for those capable of wielding new tools, are higher, so is the value of that employee to you.
The universal thing I'm hearing from friends at small-mid-size tech companies, and experiencing myself, is that there is way more work and demand for it from senior leaders than they're capable of with their current teams.
1: https://www.ciodive.com/news/tech-job-postings-hit-3-year-hi...
I think the framing started in the right path and then took a slightly wrong turn.
Both loops presented benefit from being tighter, faster. One to take a system to a “stable” (maintainable) setpoint quickly. The other to handle uncertainty.
And the additional insight about splitting the systems to better adapt to AI… we’ve described spikes for years, well before AI went mainstream.
An average unaware person believes that anything can be put in words and once the words are said, they mean to reader what the sayer meant, and the only difficulty could come from not knowing the words or mistaking ambiguities. The request to take a dev and "communicate" their expertise to another is based on this belief. And because this belief is wrong, the attempt to communicate expertise never fully succeeds.
Factual knowledge can be transferred via words well, that's why there is always at least partial success at communicating expertise. But solidified interconnected world model of what all your knowledge adds up to, cannot. AI can blow you out of the water at knowing more facts, but it doesn't yet utilize it in a way that allows surprisingly often having surprisingly correct insights into what more knowledge probably is. That mysterious ability to be right more often is coming out of "world model", that is what "expertise" is. That part cannot be communicated, one can only help others acquire the same expertise.
Communicating expertise is a hint where to go and what to learn, the reader still needs to put effort to internalize it and they need to have the right project that provides the opportunity to learn what needs to be learnt. It is not an act of transfer.
It's especially noticeable when teaching functional programming to people trained in OO: Some people's model just breaks, while others quickly see the similarities, and how one can translate from a world of vars to a world of monads with relative ease. The bones of how computation works aren't changing, just how one puts together the pieces.
However, I've seen developers who were in this field for decades, and they still followed just recipes without understanding them.
So, I'm not entirely sure, that the distinction is this clear. But of course, it depends how we define "senior". Senior can be developers who try to understand the underlying reasons and code for a while. But companies seem to disagree.
Btw regarding functional programming. When I first coded in Haskell, I remember that I coded in it like in a standard imperative languages. Funnily, nowadays it's the opposite: when I code in imperative languages, it looks like functional programming. I don't know when my mental model switched. But one for sure, when I refactor something, my first todo is to make the data flow as "functional" as possible, then the real refactoring. It helps a lot to prevent bugs.
What really broke my mind was Prolog. It took me a lot to be able to do anything more than simple Hello World level things, at least compared to Haskell for example.
Of course I was still super junior and had so much to learn, but from that point I could at least interrogate any pattern or best practice to understand why it existed and where it should or should not be applied.
Then, I met software and computer science abstractions, they all seemed so arbitrary to me, I often didn't even understand what the recipe was supposed to cook. And though I have gotten better over time (and can now write good solutions in certain domains), to this day I did not develop a "physics" level understanding of software or computer science.
It feels really strange and messes with your sense of intelligence. Wondering if anyone here has a similar experience and was able to resolve it.
Between that latter group and the bottom portion of the middle it sparked a big culture war. Eventually leading to leadership declaring that FP was arcane wizardry, and should be eradicated.
Besides OO -> Functional this applies everywhere else in Computer Science. If you understood the fundamentals no new framework, language or paradigm can shock you. The similarities are clear once you have a fitting world model.
I've been doing this for coming up on thirty years now, mostly at one large company, and I spent a significant number of hours every week fielding questions from people who are newer at it who are having trouble with one thing or another. Often I can tell immediately from the question that the root of the problem is that their world model (Naur would call it their Theory) is incomplete or distorted in some way that makes it difficult for them to reason about fixing the problem. Often they will complain that documentation is inadequate or missing, or that we don't do it the way everyone else does, or whatever, and there's almost always some truth to that.
The challenge then is to find a way to represent your own theory of whatever the thing is into some kind of symbolic representation, usually some combination of text and diagrams which, shown to a person of reasonable experience and intelligence, would conjure up a mental model in the reader which is similar to your own. In other words you want to install your theory into the mind of another person.
A theory of the type Naur describes can't be transplanted directly, but I think my job as a senior developer is to draw upon my experience, whether it was in the lecture hall or on the job, to figure out a way of reproducing those theories. That's one of the reasons why communication skills are so critical, but its not just that; a person also needs to experience this process of receiving a theory of operation from another person many times over to develop instincts about how to do it effectively. Then we have to refine those intuitions into repeatable processes, whether its writing documents, holding classes, etc.
This has become the most rewarding part of my work, and a large part of why I'm not eager to retire yet as long as I feel I'm performing this function in a meaningful way. I still have a great deal to learn about it, but I think that Naur's conception of what is actually going on here makes it a lot more clear the role that senior engineers can play in the long term function of software companies if its something they enjoy doing.
Or maybe I'm just a little bit insane. Or both.
Thesis A is something like: the value of the programmer comes from their practical ability to keep developing the codebase. This ability is specific to the codebase. It can only be obtained through practice with that codebase, and can't be transferred through artefacts, for the same reason you can't learn to play tennis by reading about it (a "Mary's Room" argument).
This ability is what Naur calls "theory". I think the term is a bit confusing (to me, the word is associated with "theoretical" and therefore to things that can be written down). I feel like in modern discourse we would usually refer to this as a "mental model", a "capability", or "tacit knowledge".
Then there's Thesis B, which comes more from a DDD lineage, and which is something like: the development of a codebase requires accumulation of specific insights, specific clarifying perspectives about problem-domain knowledge. The ability for programmers to build understanding is tied to how well these insights are expressed as artefacts (codebase structure, documentation, communication documents).
I feel like some disagreements in SWE discourse come from not balancing these two perspectives. They're actually not contradictory at all and the result of them is pretty common-sensical. Thesis A explains the actual mechanism for Thesis B, which is that providing scaffolding for someone learning the codebase obviously helps, and vice-versa, because the learned mental model is an internally structured representation that can, with work, be externalised (this work is what "communication skills" are).
Of course the model is incomplete compared to reality. That's in the definition of a model, isn't it? And what is deemed a problem in one perspective might be conceived as a non problem in an other, and be unrepresentable in an other.
Everyone should subscribe to the Future of Coding (recently renamed to the Feeling of Computing) podcast if you haven't already: https://feelingof.com/
(Second, albeit more theoretical, would be A Critique of Cybernetics by Jonas)
I try to train and mentor those that are junior to me. I try to show them what is possible, and patterns that result in failure. This training is often piecemeal and incomplete. As much as I can, I communicate why I do the things I do, but there are very few things I tell them not to do.
I am often surprised at the way people I have trained solve problems, and frequently I learn things myself.
Training is less successful for those who aren’t interested in their own contributions, and who view the job only as a means to get paid. I am not saying those people are wrong to think that way, but building a world view of work based on disinterest isn’t going to let people internalize training.
I think it becomes difficult to train the next layer up though, which is a sum-total of life experience. And I think this is what the parent poster was referring to.
For example, I read a lot of Agatha Christie growing up. At school I participated in problem-solving groups, focusing on ways to "think" about problems. And I read Mark Clifton's "Eight keys to Eden".
All of that means I approach bug-fixing in a specific mental way. I approach it less as "where is the bug" and more like "how would I get this effect if I was wanting to do it". It's part detective novel, part change in perspective, part logical progression.
So yes, training is good, and I agree that needs to be one. But I can not really teach "the way I think". That's the product of a misspent youth, life experience, and ingrained mental patterns.
"Transmissionism" is a term I've seen to describe this
"Seeing the work reveals what matters. Even if the master were a good teacher, apprenticeship in the context of on-going work is the most effective way to learn. People are not aware of everything they do. Each step of doing a task reminds them of the next step; each action taken reminds them of the last time they had to take such an action and what happened then. Some actions are the result of years of experience and have subtle reasons; other actions are habit and no longer have a good justification. Nobody can talk better about what they do and why they do it than they can while in the middle of doing it."
complexity is
not what you believe it is
please try listening
that can only be moved around,
not eliminated.
Long before the discussion of the morality of AI went mainstream, I ran into a problem with making what appeared to be ethical choices in automation, and then went on a journey of trying to figure this all ethics thing out (took courses in university, read some books...)
I made an unexpected discovery reading Jonathan Haid's... either Righteous Mind or the Happiness Hypothesis. He claimed that practicing ethics, as is common in religious societies is an integral and important part of being a good person. This is while secular societies often disregard this aspect and imagine ethics to be something you learn exclusively by reading books or engaging in similar activity that has exclusively the descriptive side, but no practice whatsoever.
I believe this is the same with expertise. Part of it is gained through practice, and that is an unskippable part. Practice will also usually require more time than the meta-discussion of the subject.
To oversimplify it, a novice programmer who listened to every story told by a senior, memorized and internalized them, but sill can't touch-type will be worse at everyday tasks pertaining to their occupation. It's not enough to know touch-typing exists, one must practice it and become good at it in order to benefit from it. There are, of course, more, but less obvious skills that need practice, where meta-knowledge simply can't be used as a substitute. There are cues we learn to pick up by reading product documentation which will tell us if the product will work as advertised, whether the product manufacturer will be honest or fair with us, will the company making the product go out of business soon or will they try to bait-and-switch etc.
When children learn to do addition, it's not enough to describe to them the method (start counting with first summand, count the number of times of the second summand, the last count is the result), they actually must go through dozens of examples before they can reliably put the method to use. And this same property carries over to a lot of other activities, even though we like to think about ourselves as being able to perform a task as soon as we understand the mechanism.
The tao that can be described
is not the eternal Tao.
The name that can be spoken
is not the eternal Name.
I say this as someone with 3 decades of professional experience. That does not make me right, please do tell me that I am in fact wrong! It does mean that I might be one of these guys with positions that should be challenged, however.
You know what? I welcome this. Explain to me why I am wrong, let's do it your way, dear youngin!
Yea, but, I have a search engine that contains all the original uncompressed training data, so I'm back on top. How we collectively forgot this is amazing to me.
> and they need to have the right project that provides the opportunity to learn what needs to be learnt.
It takes _time_. I solve problems the way I do because I've had my fair share of 2am emergency calls, unexpected cost blowups, and rewrite failures in my career. The weariness is in my bones at this point.
Agree about expertise being inseparable from the 'world model'. When someone tells us something, they're assuming that we know a certain amount of background knowledge but, in reality, we never have exactly the missing pieces that the speaker is assuming we have because our world model is different. It can lead to distortions and misunderstandings.
Even if someone repeats back to us variants of what we've told them at a later time, it doesn't mean that they've internalized the exact same knowledge. The interpretation can be different in subtle and surprising ways. You only figure out discrepancies once you have a thorough debate. But unfortunately, a lot of our society is built around avoiding confrontation, there is a lot of self-censorship, so actually people tend to maintain very different world models even though the surface-level ideas which they communicate appear to be similar.
Individuals in modern society have almost complete consensus over certain ideas which we communicate and highly divergent views concerning just about everything else which we don't talk about... And as our views diverge more, it narrows down the set of topics which can be discussed openly.
Most smart juniors have no problem with learning. Perceptual exposure and deliberate practice works almost mechanically. However, if someone can't tell you what examples you should be exposed to, you'll learn crap.
I'd love to talk more live. I think I have some ideas you'd be interested in. Find me in my profile.
This is also why average people with little time to commit find it hard to realize the importance and depth of AI. It's a full on university education exploring those.
Great thread.
Guess which of those people say education is a waste of time.
Basic Education is about bringing the average up, make sure everyone can read and do basic math etc. Beyond that it's the same thing just at a higher level. Most people who graduate with a bachelors of computer science or similar won't be great programmers. That's why we call them great, they're better than the rest. Most people don't have greatness in them. No amount of education can change that. Those who do have what it takes will naturally succeed because that's who they are. They will study and do the required material and when they're done with that they will spend their free time learning even more because they want it. Some will do so even without a formal education but I don't think that's the case for most.
If you manage to go through 3-5+ years where your only job is to learn stuff, without learning stuff, then you have no one to blame but yourself.
And if you can do it without going to university, go ahead. I'm pretty sure the failure rate for that path is a lot higher but it's certainly an option.
The best educators I had had exactly that approach: you sometimes start with theory, but other times with challenges which make you feel the difficulty, and understand the value of the theory you are co-developing with the educator (they just have the benefit of knowing exactly where we'll end up, but when time allows, they do let you take a wrong turn too). Even if you start with theory, diving into a challenge where you are allowed not to apply the learnings should quickly tell you why the theoretical side makes sense.
As with everything in life, great educators are few but once you have them, you can apply the same approach yourself even if the educator is unable to steer you the right way.
If you never received this type of education, then what you received could arguably be called a waste of time.
Actually, maybe even worse (not directed at parent) - I think some "seniors" have a stick so far up their err keyboard, and think they are so wise beyond words that they refuse to share their "all knowing expertise" with anyone else as a form of gatekeeping or perhaps fear of being "found out" (that they are not actually keyboard "Gods").
Really though, just wright shit down even if the first draft isn't great. Write it down, check it into the codebase.
> “Do we really need that?” > “What happens if we don’t do this?” > “Can we make do for now? Maybe come back to this later when it becomes more important?”
as with experimenters. Every system is different, every product is different. If I were building firmware for a CT scanner, my approach towards trying out new things would be different than a CRUD SaaS with 100 clients in a field that could benefit from a fresh perspective.
There are definitely ways to have eager/very open seniors drive systems into hard to get out corners. But then there are people that claim PHP5 is all you need.
A rewrite?
I recall a few times everyone promised, if this gets promoted then we will rewrite it from zero. Never happened.
The article touches on responsability, accountability. There is none for risk taker. By definition. You have a crazy idea, you rush it out, you hope clients bite. You profit. It's not even your problem how to make it work, scale, not cost more to run than we sell it for.
The loop on the right. There are companies, two of them are very popular these days, they took it to an extreme. You ship something fast, and since it only scales linearly you go raise money. Successful companies, countless users, some of them even pay. Who's to blame? The senior developer, or simply someone reasonable who asks, how's that sustainable, what's the way out of this? Those are fired, so whoever's left is a believer.
So it's not like I have nothing to share after 30 years of experience in the industry, I just have nobody to share it with.
One of my favorite .sigs was:
I hate code, and want as little of it as possible in my software.
One of the things that happens to "avoiders," is that they get attacked for being "negative." It can get career-ending, when the management chain is the "Move fast and break things" type.
I just stopped offering suggestions, after encountering that crap a few times, and learned to just quietly make preparations for when the wheels fall off.
I have spent my entire adult life, shipping, and shipping means lots of "not-shiny," boring stuff. But it gets onto shelves, and into end-users' hands. I was originally trained in hardware development, where mistakes can't be fixed with an OTA update. It taught me to "play the tape through," and make sure that I do a good job on every part of the project; which includes a lot of anticipating problems, and designing mitigations and prevention.
The problem I see is management is playing games with not talking about how much money is available, what the real timelines are, etc - because they fear the people contributing will leave before the critical moment and so people keep making stupid decisions in that context and then you all get to get a new job.
There exist people who's jobs depend entirely on rolling out new features, or apps of some sort, and having them show up in some form of company metric. If the senior developers says it's a bad idea, those people won't listen, or won't care. Their job is on the line.
> Need to build a whole new feature to test it? Have you tried putting a button in the existing UI and seeing if people click it?
It feels like engineers are collectively feeling the pain now that product has decided that engagement of mental faculties is no longer necessary on their behalf; just build it and figure out the user persona and utility later...if ever. What used to be a process of taking the time to understand the domain, the user, and how the product fits into some process has been tossed out the window; just ship whatever we think some imaginary user wants and experiment until we succeed.
It creates the exact problem that OP talks about: every random feature that gets vibe-coded becomes a source of instability and risk; something that can then only be maintained via more vibe coding because no one has a working mental model of the thing.
There are other properties such as, maintainability, scalability, reliability, resilience, anti-fragility, extensibility, versatility, durability, composability. Not all apply.
Being able to talk about tradeoffs in terms of solution spaces, not just along a single dimension, is one of what I consider the differentiator between a senior and staff+ developer.
As this kind of person, it can be alienating in some teams / companies.
What I've found works best is to convey how the added complexity will affect non-engineers. You have to understand the incentives and trade offs though, and sometimes it's better to take the loss.
If you have the fortune of sticking around with the same leaders for awhile, a few rounds of being vocal, but compromising, will work in your favor. When that complexity comes back around to bite them in the way you described, you will earn some trust.
In my experience the solution proposed will rarely result in a less complex solution. Quick MVPs have the tendency to stick around. As soon as a customer starts using some product or feature, the cost of pivoting goes up. If you wish to experiment, do it on a segment.
AI is actually quite awful for prototyping, because it makes it far too easy to add random crap to your "prototype" without any specific intention. This quickly transforms the prototyping process from something that's high-level and geared towards building the mental model of the real system into something akin to copy-editing a random piece of software without any coherent mental model involved. Moreover, prompting allows to to glaze over some essential complexity of the task without getting any notions of the scope of the effort of actually doing it. In other words, people end up failing to make necessary decisions and simultaneously get bogged down with unnecessary ones.
In short, fast feedback loops are only useful if there is actual feedback involved.
None of the things I can think of have anything to do with avoiding problems.
To some degree, having 5+ agents working on different projects is similar to leading a team of 5+ people. The skills translate well.
The senior is also able to understand what the agents do, review and challenge it. Juniors often can't.
And finally, the senior has a deeper understanding of what the business and problem domain are, and can therefore guide the AI more effectively towards building the right thing.
If anything, quite often it's introducing more problems, because we know we'll run into them and they need to be addressed.
AI is sometimes quite lazy and refuses to solve the hard problems (sometimes making funny excuses like it would take weeks) until you make it explicit that it's important they are dealt with.
> “Sure, I’ll have the Speed version ready in 3 days. Then the Scale version in about 6 weeks.”
> They get what they want, speed and momentum. You get what you want, observation and design.
Except that 6 weeks is now blocking the next thing and you'll be pushed to drop it. So this doesn't really solve anything.
I was kind of hoping at the end they'd suggest getting the non-developers involved more so they can experience the pain points they're creating. Not entirely sure how that would work though.
As you might imagine, a lot of these ideas fell by the wayside but we had to develop them in full.
You see the problem immediately. Sales/marketing didn't do their job sussing out what X is and wastes dev time with Ys. And worst of all, write once, support forever. Each one off Y has to be maintained for the special snowflake customer that uses it. None of the Ys actually work well for all the customers with problem X so you end up drowning in "technical debt" spent to create them all.
If your marketing department leads the company, I've discovered the best option is to just quit. Go find a job at an engineering company.
Non-developers have no clue WHAT they want, then know WHY they want it. The why is much more important to know, because the requestor has no clue how software works and imagines bad solutions.
There's ways to navigate it.
PING
Middle developer: PING constructs and sends ICMP packets to an address
Senior developer: what machine, what OS?
Junior manager: Don't care, ask a techie if you need to do something technical
Middle manager: Ask <techies name> about it, I know he has great experience with it
Senior manager: PING is used to check if a host is reachable by a network
Senior developers fail to communicate their expertise, because that expertise is developed and formed by asking more questions than answers, and managers fail to understand the capabilities of "their techies", because managers see question-asking techies as counter-productive, and attempt to route around them. Managers only want answers, developers know the value of asking deep questions.
Thus, AI.
(BTW, PING is a command that produces a distinct sound on the Oric-1/Atmos computers, and it is thus an Onomatopoeia.. I know this, because I am a Senior Oric-1/Atmos Developer who knows what lies at #FA9F, how it works, what the 14 bytes are for, and so on.. because I once asked the question, "how does PING go 'poooinnng' but ZAP go 'zap'?")
AI: <asks billions of questions in a second>PING is ..
1. I am discouraged or forbidden from devoting time to communicating my expertise; they would rather use it. Well, often, they'd rather I did the grunt work to facilitate the use of my expertise.
2. Same, but devoting time to preparing materials which communicate my expertise.
3. A lot of my expertise is a bunch of hunches and intuitions, a "sense of smell" for things. And that's difficult to communicate.
4. My junior colleagues don't get time off their other duties to listen to "expertise sharing", when it does not immediately promote the project they're working on.
5. Many of my junior colleagues lack enough fundamentals (IMNSHO) for me to share all sorts of expertise with them. That is, to share B with them I would need to first teach them A, and knowing A is not much of an expertise; but they're inexperienced, maybe fresh out of university.
6. My expertise may only be partially or very-partially relevant to many of my colleagues; but I can't just divide the expertise up.
7. For good reasons or bad, I have trouble separating my expertise from various ethical/world-view principles, which fundamentally disagree with the way things are done where I'm at. So, such sharing is to some extent a subversive diatribe against the status quo.
8. My expertise on some matters is very partial - and what I know just underlines for me how much I _don't_ know. So, I am apprehensive to talk about what I feel I actually don't know enough about - which may just result in my appearing presumptuous and not knowledgeable enough.
9. My expertise on some matters is very partial - and what I know just underlines for me how much I _don't_ know. So, I try to polish and complete my expertise before sharing it - and that's a path you can walk endlessly, never reaching a point where you feel ready to share.
10. Tried sharing some expertise in the past, few people attended the session, I got demotivated.
11. Tried sharing some expertise in the past, few people were engaged enough to follow what I was saying, I got demotivated.
12. Shared some expertise in the past, got a positive feedback, but then those people who seemed to appreciate what I said did not implement/apply any of it, even though they could have and really should have.
Now, with so-called AI they will mostly slap something kinda working in few days and then maybe get hacked or double invoice some customer from time to time... They will learn of those problems the hard way. Or maybe they will not because it will be mostly working emailing system and nobody will care if it will loose 2% of the emails because of some bug.
Nevertheless, either the Stable version, Scale version of the software will never happen or will be looked like not necessary or it will became a thing after catastrophic failure.
Anyway I do not think it will be like that, everybody cares about speed and money and making money quickly without an effort is the ultimate unicorn entire world is after.
Those complaining developers just stand in a way.
People don't want to be judged in the introduction of an article, based on how they like to approach their literal dayjob. It's a weird jab.
FWIW though the idea about a "speed" product and a "stability" product isn't new. We used to call it "prototyping". I don't know when/how that disappeared from the collective consciousness. "have a space where we can build things fast with horrible practices" isn't some AI era innovation, it's what smart companies have done for decades.
> What if we had one system just for speed?
Like a beta? It would take incredible discipline from the business and customers not to consider that production software and demand 99.99 uptime and bug free.
The company provides (offer | service) to the (market | user) and receives (feedback | payment).
The service IS the offer, the userbase IS the market, and payment IS the feedback signal.
Right?
EDIT - expanded on original comment to add:
The author's point might be lost on me but seems to be that framing things with one of those sets of labels vs the other may correspond to use of "complexity" vs "uncertainty" as the element targeted for reduction, and choosing those labels carefully in turn correlates to "senior" devs' persuasiveness in prioritization battles with product owners. To which my response would be, "maybe?". (shrug)
I'm not a copywriter by trade but I care about words and may have just been nerd-sniped.
The company is offering potential new services to current and potential users in the market and getting feedback on how valuable those new services might be.
That said, I think everyone will agree both extremes are failure modes - moving too fast and not having things work right, or building things too correctly and never building something people want.
IMO the two stereotypes in the article generally hold true, and the job of each in a healthy company is to present the trade-offs, and collect enough data to experimentally validate. And when you disagree, let the decision maker (CEO) decide, but disagree and commit
What’s missing from a lot of the “AI replaces developers” discussion is that generating code is only a tiny part of operating production software. The hard part starts after deployment: debugging, understanding cascading failures, maintaining consistency, and safely evolving systems over time.
Feels like the next wave of tooling won’t just be “AI that writes code faster”, but AI that helps absorb and reduce operational complexity after the code exists.
When I started around 20 years ago, my junior dev experience was pretty harsh - I was taught, not always in a correct or respectful manner, to do this and not to do that. Overall though, it was absolutely useful and formative. Senior engineers are rarely abusers, they communicate real issues, better or worse, and it was on me to figure out why and how to work the right way. Also we were raised in a pretty receptive attitude to the "old" technology - from Tcl and Smalltalk to Ada, Perl, etc. It was admired classics rather than just old shit.
Surprisingly, this didn't translate too well to my experience when I found myself in a mentoring position. Starting from 2015 maybe the situation changed. Newer generation of devs felt much more entitled to social games, higher salaries and opinions rather than authentic engineering interest and therefore my experience.
No amount of structured communication would change that, even the cold pressure of production failures and very specific poor management feedback normally doesn't work. They're also more lenient to prod screw-ups, and often use "everyone can make a mistake" excuse for excusing even more mistakes. The thing is, most of them don't want to hear for any reason.
As many of my peers I learned humility and accepted that as is, only using my advantage in expertise when it comes directly to my responsibility territory, and to avoid a hassle imposed by my eager younger teammates, like I usually parse prod logs and settings with command line while the younger guys trying to push through loki/grafana query limitations.
I'm fine and safe, and my job is no less secure, I guess, because someone has to fix bugs etc. The companies less so, but as long as they don't care why would I.
It will be interesting to see this generation wiped off by the next one. I guess they shouldn't be in a very good shape, because the foundation they built upon (namely quickly changing libraries and language supersets like React/TypeScript/some JVM flavour/and I hope Kafka) will be replaced by the next tech fashion.
> Ah, baby, this is my senior developer. The avoider, the reducer, the recycler. They want to avoid development as much as they can.
There are times when this is good, there are times when actively trying introduce an improvement is the best way forward. A good senior is able to recognise when those times are.
Now combing through analytics to determine whether or not what we did was actually good? Less warm and cozy.
This is what I was thinking - I'd say the biggest step up a developer can make is to recognize that sometimes you need a bit of one approach, sometimes a bit of another one.
Sometimes minimalism is the way, and you need to wonder if the pain, workload or lacking capabilities and features are problematic. Or, sometimes adding the smallest possible thing is a good way, as long as we don't paint ourself into a corner and enable learning and accumulating information of what we actually need.
Sometimes buying a thing is a good way, if you can find a good vendor and a tool fitting your use case and especially if the effort of doing it on your own is high. This commonly occurs in security, because keeping up to date with the ongoing vulnerability and threat landscape can be a full job on its own.
And sometimes adding something bigger is the way, if the effort of maintaining it are less than the effort and pain incurred by not having it. Or if we can ramp up the effort of the thing incrementally, while reaping benefits along the way. This can be validated often by doing a small thing.
What the AI will do in my opinion is to push the bar more in this direction. Cozily hacking CRUD-Code in a web server together most likely won't be enough in a year or two for the average development job.
I read the above as "avoid development that increases complexity needlessly" — and often, there is a desire to overcomplicate something that can be much simpler because the understanding is lacking.
"As much as they can" does not mean trying not to do any work, but trying to simplify the work where it achieves desired outcomes, and just about! This frequently means doing the improvement today.
preventing the unnecessary changes can help you get the political capital in your org to push through the changes that really need to happen.
I bet there's money to be made for building a drop-in to either of those two that requires less memory, would save companies a bundle, and make other companies a bundle as well.
This do not mean we don't develop new product and services, it just means when we do so, we find the path of least overall entropy, it also applies to operations and tech debt reduction.
premature optimization is root of all evil
The qualities were highlighted because they can all lead to better stability.
> Yes, yes, of course this is simplistic.
It's an example, put to the extreme, to clearly communicate the ideas. As all things, the golden mean applies, as I understand the article argues for:
> the design of the 'Scale' version is influenced by what worked and what doesn’t work in the 'Speed' version of the system.
... All of them?
On the other hand, almost any business problem can be solved in a reasonable way that doesn't send your system through any terrible one-way doors if you zoom out enough and ask enough whys. Of course not every place allows engineering to do that, but the ones that don't aren't able to retain senior folks because they will just go somewhere where their judgment is valued. Sometimes technical debt is the right thing for the business, but sufficiently senior engineers can set things up so there is always a way out. But what you can't do is uphold the purity of the system above the business problem. The systems are paid for by the business, so if you lose sight of that then you've lost the plot and the basis for your influence.
Old quote: "There is nothing so permanents as a temporary hack."
At this point Zig implementation of Bun seems like one written to throw away. And it happened only thanks to AI.
We were highly autonomous team though and hardly had cadence complains. But mostly because the all other departments were lagging. Except marketing, marketing always has "ideas".
That's not to say that my first pass that I show people is ready to go into production, but I build the PoC from the beginning with the idea that it _is_ going into production and make sure I have a plan to get to production with it while I am working on it.
Why would you do that though? If you have a working 'prototype' that's handling the demand, has the required features, and doesn't really need to be rebuilt (except to appease the sensibilities of the developers), why would you spend time and effort on that? That makes no sense. The fact it's a prototype or a 'proof of concept' is essentially irrelevant if you can't enumerate what the actual problem with it is.
I work with a bunch of teams that complain that they're mired in tech debt all the time, and complain that it's a huge risk and it slows them down. Except I can see our incidents log and there aren't many incidents and none that can be attributed to running risky code in prod, I have our risk register that has no 'this code is old and rubbish and has past-EOL dependencies on it', and no team has ever managed to articulate how or even how much the tech debt slows them down. They shouldn't really claim to be surprised that no one wants them to spend time 'fixing' a problem that apparently has no impact.
I've also seen the opposite case where a team spent months refactoring an app that they wrote before it launches. They wrote it, then decided they could make it 'better', and spent loads of time reworking most of it before it launched. All the value was delayed because they decided they didn't like their own work. And obviously the leadership team were pissed off about that, and now there's very little trust left.
There should be a good conversation about delivery of work between teams and stakeholders or no one will be happy, but if that isn't happening the stakeholders will always win.
You can get a few feet closer to the moon by building a treehouse, but you still can't turn it into a spaceship.
IMO it is a bit arrogant to assume it is more important to engineer a better version of a thing rather than make money quicker and cut corners. In essence it is better to have a problem which is about how to scale a new product because it got traction rather than solve a problem how to sell more copies of already scalable thing.
Rewrites require an existential-level threat to pursue and should never be taken lightly. They must solve a real verifiable need, backed by real world data. Rewrites for rewrites sake or some lofty or nebulous goal of "better" or "more maintainable" code are doomed to fail and a waste resources.
I've seen the worst of it, from your average monoliths with no separation of concerns to 1000s of lines of self-modifying assembly in dead architectures with no code comments containing critical business logic, etc.
The main rule is to not to bite off more than you can chew, which if I'm being honest you really only learn from fucking up or watching others fuck it up.
Hackathon and overnight oncall fixes ABSOLUTELY should be rewritten or production-hardened, but they very often are not.
A less experienced dev suggested using "AI magic" to replace a URL validator. I protested, suggesting a cached fuzzy match solution (prepopulated by AI)... and no one cared. Now the AI model has been suddenly turned down, and our system is broken. We're going to have re-validate the whole system.
A younger developer who got promoted over me tried to write a doc on possible ways to fix it. He said "hey Dan, can you help me with this?" He got promoted over me because the way to get ahead is to write docs and have meetings, not do things sensibly. Now he's trying to use my work to demonstrate his leadership.
No one cares. The more I offer better solutions, the more it's a threat to less experienced developers. Things mostly work so my manager doesn't care. There's probably better ways for me to have handled things, but it's so exhausting fighting the nonsense and I just want to write good code.
Looking deeper into it: these people don't understand the underlying foundations anymore. Just keep building fast, without building proper mental models (that would take time).
Our work is largely very difficult to understand to outsiders, we need to write docs and have meetings to show what we have done. It's part of the job, and yes, if you don't do that, it doesn't matter how fantastic the software is that you wrote (sadly).
To me, young people just don't seem to know, or want to know, that information and knowledge can be gained from a person. It's the arrogance of youth x100
They have a supercomputer in their pocket/on their desk, and an AI that knows 'everything'. I can't imagine what it's like being a teacher right now.
How's your AI going to explain the office politics? The CTO's opinion on things? Talk about recent outages and learnings (details of which are not often on blogs)?
They think all they need is knowledge and facts and none of history, politics, communication etc
I think a lot of is that an AI or Google search won't challenge them, push them, disagree with them - and that's comforting to them, and more desirable than the learning that could happen
It's just basic game theory, and you see it everywhere. However, it's so annoying in the workplace when your two options seem to come down to try to dominate or be dominated. Especially if you care about quality code and don't care for meetings.
As far as I'm concerned, I think I have to make peace with the fact that if I don't play the game, I am going to be managed by people who don't know what they're doing. But neither option seems particularly good. Should I try to bury my ego and influence from below? Should I work harder and try to climb the corporate ladder? I'm still not sure.
seriously. it kills me to have so much knowledge and expertise that few people appear to care about if not downright hate me for wanting to pass it on to others as it appears institutional knowledge does not have any value these days
It's simply the case that the supply of "experts" wanting to share "expertise" vastly eclipses the demand by several orders of magnitude.
I think there's a business somewhere, where you get paid to listen to "experts" and they get to feel better about themselves. It's a win-win.
So if people don't perceive you as an "expert" and dont go to you for answers, you simply do not register as one or they have a rather high bar which requires observable undeniable artifacts (and I don't mean credentials, I mean software) and competition is rather fierce - there's simply overproduction of people who think they are "experts" and thus you have to give unmistakable symptoms of being one to register.
Companies have outlandish hiring practices. They want juniors who already know everything. That's why admitting that you don't know something is seen as showing weakness to the company in the eyes of a junior. Also, not knowing things will actively keep you from getting promoted.
I'm sure it's not like that everywhere but it's juniors playing the corpo game.
Whereas juniors are eager to chat, have lunch with you , and share what they’re working on, the seniors are guarded and solitary.
Maybe that’s just my workplace though!
And yes, the office is important.
Orgs get what they measure for. If your team values that sort of interactivity and support, it will ... observe it, measure it, and hire for that sort of person. I've seen groups evolve towards that, and they've been great, but it doesn't seem to be a default - most groups/orgs have to work towards it and and keep working at it.
That said, I completely agree. I learned most of what I know from being in the same room with senior developers and asking questions. Something that just isn't happening these days.
"It takes two to tango" i.e. junior developers must first put in some effort and then proactively seek out seniors with expertise.
It may be a cliche, but a truism nevertheless; viz. the juniors are simply not interested in putting in the necessary time/effort to gain knowledge systematically. They want everything to be quick, easy and handed to them on a platter.
I think the main reason for this is; there is just too much out there to learn and everything is being propagandized as being the most important and most indispensable; This swamps the juniors and hence they feel lost and try to keep up with everything which is a fool's errand.
Juniors need to keep the following in mind;
1) Change their learning mindset as follows; - Browse a lot, Read a subset and Study an even smaller subset.
2) Always focus on the essentials and not on the frills. This is determined by your specific goals/needs.
3) Be okay with not knowing everything. Do not base your self-worth on others evaluation of you.
4) Do not compete with others. Do the best you can and always improve on your yesterday's self. As the adage goes "drops of water falling, if they fall continuously, can bore through iron and stone".
5) Be confident in your own intelligence. As Sherlock Holmes said "what one man can invent another can discover". What might seem impenetrable in the beginning will over time become clearer and easier when studied regularly.
6) Everything is dependent on Self-Effort modulated by Timing, Context, Means Employed and finally Random Chance (i.e. lady luck). Manage the last by factoring in its payoffs as part of your self-effort itself (i.e. hedging). Focusing on the above five parameters before starting on anything will guarantee success.
7) You can always short-circuit your studies and gain knowledge quickly by asking seniors with expertise to teach you. Your attitude and way of approach is very important here i.e. you must be sincere and committed.
Of course, he turned in his notice shortly after I arrived, because he had found his successor. So, that didn't work out so well for me.
- Juniors are discouraged to ask for mentorship because they are under pressure to appear competent
- Juniors have internalized from bad experiences that seniors are not to be disturbed
- Juniors grew up in a world where nobody modeled mentorship as a possibility for them; a CS major probably learned async, online, parasocially, without much 1:1 face-to-face interaction
- Juniors don't know what they don't know just yet-- and it doesn't always work well for someone to try and teach them explicitly-- but once they figure this out they'll be more interested in reaching out
I also believe that some of seniors experience is flesh-level resilience. I'm no smarter than when I joined the industry, I just got used to being in the trenches, how to handle my own psychology, how all the easy-looking things are not and how the horrible ones aren't either.. I could explain this in detail to any junior, but until they're on the minefield it won't mean much.
Honestly I have the feeling that this is often insecurity. It's easy to feel uncomfortable if you think you don't follow along.
Another issue is that juniors usually experience culture shock on their first jobs. So they more or less isolate and do thing how they learned it.
I've been a mentor off and on for the last few decades, and I've been really lucky to have some strong mentees. Some I've followed for a better part of a decade and are crushing it out there. All I can really say is that they're out there, sorry I don't have any more helpful to say around how to find them etc. I'll mull on that for a bit..
“Complexity” understood as what’s gonna make development on this system fly easy and fast for the next 10 man-years de facto means side steps when naive approaches would charge straight ahead.
Tortoise and the Hare… the urge to hurry up and burn hard the first two weeks (low hanging fruit, visible wins, MVP!), resulting in ever decreasing momentum due to immature design and in-dev maintenance needs is befuddling to me. So much “faster” for weeks, and it just meant the schedule slipped 6 months.
Well said. In Kent Beck’s Tidy First, it explores the slow process that can be summarized by this except from his substack [0]
“Valuable” lives on 2 axes:
Features—what the code does now.
Futures—what we can get the code to do once we learn the lessons of this set of features.
I have also seen projects go badly because the eng was trying to be perfect upfront. Whereas quickly getting to an MVP and then iterating tends to go better.
And, the Cynefine Framework defines “complexity” a bit differently than the intuitive way it’s often used.
The simple domain is a single dimension. The complicated domain is a system of factors. I think when most people say “complex”, they are really talking about what Cynefine labels as “complicated”.
The Cynefine complex domain is not so easily solved or reduced. It has emergent behaviors. The act of measuring tends to perturb the system. No single solution will ever solve something in the Cynefine complex domain, because the complex system will shift behavior, making solutions that worked before start working against it.
Examples are ecosystems and economies. Software systems tend not to be complicated, not complex, until you start getting into distributed systems.
One of the key insights of Cynefine is understanding that each of the domains has its own way of solving things and that often times, people use solutions and methods from one domain to solve problems characterized by a different domain.
You don’t solve problems in the complicated domain with methods from the simple domain. And you don’t solve problems in the complex domain with methods that work for complicated domains.
I think these people also need to learn that, in the eyes of the customer, a sandwich with a candle is in no way comparable to a birthday cake.
> This will allow for us to deploy the feature in only X days supporting Y use case with Client W who has been complaining about this shit for Q months now.
Arguments like:
> We should do Z because it would provide future extensibility.
> Z could eventually enable some novel platform capabilities.
> Z is easier to unit test.
Are much less likely to succeed in the business contexts that I have experienced so far.
That can work too, e.g. when demonstrating the pain a customer will experience when something complex is poorly designed (like some b2b workflow), but it's less visceral than telling your internal stakeholders all the extra work they'll have to do if it's rushed. Even the best of your peers are a bit selfish. The business side has a lot of incentives around quick turnarounds so it's easy to overlook the downside.
Imagine such a scenario. You're in healthcare and working on a feature that will add new data model for some kind of clinical information.
You could say:
> This will allow for us to deploy the feature in only X days supporting Y use case with Client W who has been complaining about this shit for Q months now.
Yeah that very well may prevent W from churning, though hopefully you think about how it will affect other clients too.
Or, you could say
"If we get this data model wrong, and the value set is ambiguous, you (product/sales/cs) will have to reach out to every single customer and clarify what they meant by x/y/z if we wish to migrate it with any degree of accuracy in the future."
That's drawn from experience but I'm sure there are a lot of parallels to that in other industries for any kind of data. Migrating data is a pain in the ass for everyone, but often it can be the people pushing for a quick solution that suffer the most when that goes wrong.
This kind is stuff is why commission structures should consider churn / residuals. Bad incentives make for hastily made decisions.
Building trust is yet another quality of a good senior. By that I don't mean to be buddy with the CEO but earning trust from everyone by making good decisions, arguments and delivering as promised. Even giving a jr a warning and let him fall flat is a good trust building exercise.
My guy LeCun believes in deterministic systems describing reality even more than LLMs. He is literally a symbolic logic die hard.
Who wrote emails in haiku
It got old quickly
....
Sorry, I couldn't resist!!
Very cool
To keep it all in a clump
Than spread it about
Is the improvement likely to reduce maintenance overhead (and thus cost)? Or improve performance allowing for fewer services running (and thus reducing cost)? Or reduce bugs that force people out of a workflow (eg in an online shop, thus fixing it increases sales)?
Or if it’s just tech debt then use Jira (etc) to your advantage and talk about the number of tickets you can close of this sprint due to this engineering initiative.
If the development team and product teams goals are largely aligned then the problem with engineering initiatives is just how you explain them to the product team.
No real value is this comment, I'm just happy to share a moment over the brain-fuck that is Prolog (ironically Brainfuck made a whole lot more sense).
The problem is, as is evident by this article and thread, it's difficult to measure (and thus communicate) expertise, but it's really easy to measure years of experience.
The product managers proposing things have their reputation tied up in them so a major feature is a chance at fame for them.
It's as if everyone gets "a turn" at using the development department once in a while and they want to make the utmost of it, knowing that the instant their feature is "finished" the spotlight will be gone from them for months.
This is a valid why for a feature though. Clients want it.
Now, the concern here is do the clients want it, or does sales/product THINK the client wants it?
In a world where people (stakeholders, Product, and dev teams alike) want the prototype to be the full set of MVP features, this is not true.
I kind of get it as you have expressed that promotion is not your goal. However, organizational influence comes through promotion as your org & only those with influence at your org can change that.
What do you think would be a better system, that decoupled promotions from influence & enabled you to provide your experienced opinion without getting into management?
I just hate it because I find meetings draining.
The use of “complexity” in terms of systems theory in comparison to “complicated”, is often misunderstood.
I also agree that it’s a really good framework for evaluating problems and then making decisions on potential solutions because each has its own set of approaches.
Small nick pick. It’s “Cynefin” not “Cynefine”. The word is Welsh (Cymraeg). Roughly pronounced ke-ne-fin.
> "Software systems tend not to be complicated, not complex, until you start getting into distributed systems."
these days so much software is "distributed systems".
For example, at a level of scale, Kubernetes start having emergent behavior.
On the other hand, it doesn’t take much to produce a complex system. The Boids simulation is a complex adaptive system in the form of a flock, yet each member of the flock concurrently follows only three basic rules.
I think complexity is a byword for 'unintentionally complicated' here.
On the internet you can learn from and sometimes interact with the best of the best, so the barrier of entry for what constitutes an "expert" is rised much higher.
I still vividly remember reading a z80 instruction set manual on a rainy day during summer vacation by a lake as a kid (maybe 14?)--writing my own assembly by hand in the margins for fun. TBH I probably still have that exact manual in storage somewhere. Had a green stripe down the front edge/binding iirc.
Back then I easily met folks like myself out there on the net, including many kids younger and smarter than me. It was awesome.
I do hope that some form of that 'net lives on in spirit somehow, given that the Internet I knew has largely fallen to corporate interests.
Now that I have my own kids, it's been painful to watch them have such an utterly different experience than I did.
Their Internet is based entirely on consumption and dark patterns designed to capture their attention, while providing nothing (to them) in return besides a dopamine addiction and body dismorphia.
It is a problem as old as human civilization that the old overlook that society itself changes and instead lament the willfulness of the young in abandoning the old ways.
It isn't like young people grew up surrounded by examples of mentorship and arrogantly chose otherwise. In the internet age 1-on-1 face-to-face instruction is rare. I feel really fortunate that I caught the tail end of it.
I think younger people have maybe thrown the baby out with the bathwater, and you need some discernment on whose advice you can value and trust. But ive just been in many situations in my life where ive asked for advice and its just been total shit.
"Wisdom of the elders" is overrated when society changes so rapidly, and not all the adults you know are the insightful village shaman.
I recall asking my grandfather what is was like to live through the JFK assassination and just recieving something to the effect of "oh yeah that was crazy and bad, i remember seeing it on the news." follow up questions produced no further insight. So you come to the conclusion, why bother with that when you can just read a book about the topic.
Innovation can reduce pain though, if the current pain is strong enough. A stable stream of failures in production can be the kind of "stability" you want to disrupt.
A complete stability is death.
Read why programming languages have the structures what they have. Challenge them. They are full with mistakes. One infamous example is the "final" keyword in Java. Or for example, Python's list comprehension. There are better solutions to these. Be annoyed by them, and search for solutions. Read also about why these mistakes were made. Figure out your own version which doesn't have any of the known mistakes and problems.
The same with "principles" or rule of thumbs. Read about the reasons, and break them when the reasons cannot be applied.
And use a ton of programming languages and frameworks. And not just Hello World levels, but really dig deep them for months. Reach their limits, and ask the question, why those limits are there. As you encounter more and more, you will be able to reach those limits quicker and quicker.
One very good language for this, I think, is TypeScript. Compared to most other languages its type inference is magic. Ask why. The good thing of it is that its documentation contains why other languages cannot do the same. Its inference routinely breaks with edge cases, and they are well documented.
Also Effective C++ and Effective Modern C++ were my eye openers more than a decade ago for me. I can recommend them for these purposes. They definitely helped me to loose my "junior" flavor. They explain quite well the reasons as far as I remember.
`[state_dict.values() for mat to mat2 for row for p to p/2]`
Or similar, where data flow is 1->2->f(2)->3->4->f(4). Where right now it is this lovely mess with one more repeating term:
`[p / 2 for mat in state_dict.values() for row in (mat 2) for p in row]`
Where the flow is f(4)->2->1->3->f(2)->4->3
This is not just a Python list comprehension problem obviously. The simple for… in… has a similar problem. It’s only better, because the first term `p/2` is at the end.
I've always had trouble internalizing the "physics" of physics or chemistry, as if it were all super arbitrary and there was no order to it.
Computation and maths on the other hand just click with me. Philosophy as well btw.
I guess I deal better with handling completely abstract information and processes and when they clash with the real world I have a harder time reconciling.
"Chemical bonds fill the electron shells, which is why we have CO2. But don't worry about why carbon monoxide exists."
"Here's a formula to figure out the angle between atoms in a molecule. But it doesn't apply to H2O, because handwavy reasons. Just memorize this number instead."
Students don't gain an understanding of the subject, because the curriculum doesn't even try to teach it.
math and logic are closer to a basis for software abstraction - but they were scary to business people so a "fake language" was invented atop them - you have "objects" that don't actually exist as objects, they are just "type based dispatch/selection mechanism for functions", "classes" that are firstly "producers of things and holders of common implementation" and only secondarily also work to "group together classes of objects"
I do not think OOP ever really worked out well as can be evidenced by it no longer being as popular and people having almost entirely abandoned "Cat > Animal > Object" inheritance hierarchies.
OOP didn't really take off either, but mostly because it is hard to optimize and impossible to type.
I agree so much with this. It's why I feel so stifled when an e.g. product manager tries to insulate and isolate me from the people who I'm trying to serve -- you (or a collective of yous) need to have access to both expertise in the domain you're serving, and expertise in the method of service, in order to develop an appropriate and satisfactory solution. Unnecessary games of telephone make it much harder for anyone to build an internal theory of the domain, which is absolutely essential for applying your engineering skills appropriately.
Another facet of this is my annoyance at other developers when they persistently incurious about the domain. (Thankfully, this has not been too common.)
I don't just mean when there are tight deadlines, or there's a customer-from-heck who insists they always know best, but as their default mode of operation. I imagine it's like a gardener who cares only about the catalogue of tools, and just wants the bare-minimum knowledge to deal with any particular set of green thingies in the dirt.
Edit: The main role of PM is to decide which features to build, not how those features should be built or how they should work. Someone has to decide what to build, that is the PM, but most PM are not very good at figuring out the best way for those features to work so its better if the programmers can talk to users directly there. Of course a PM could do that work if they are skilled at it, but most PM wont be.
Even the most verbose specifications too often have glaring ambiguities that are only found during implementation (or worse, interoperability testing!)
In practice, it isn't.
Product designers have to intuit the entire world model of the customer. Product managers have to intuit the business model that bridges both. And on and on.
Why do engineers constantly have these laughably mind blowing moments where they think they are the center of the universe.
Software people do what they do better than anyone else. I mean obviously! Just listening to a non-software person discuss software is embarrassing. As it should be.
There's something close to mathematics that SWEs do, and yet it's so much more useful and economically relevant than mathematics, and I believe that's the bulk of how the "center of the universe" mindset develops. But they don't care that they're outclassed by mathematicians in matters of abstract reasoning, because they're doers and builders, and they don't care that they're outclassed by people in effective but less intellectual careers, because they're decoding the fundamental invariants of the universe.
I don't know. I guess I care so much because I can feel myself infected by the same arrogance when I finally succeed in getting my silicon golems to carry out my whims. It's exhilarating.
If the programmer gets to intimately understand the user's experience software would be easier to use. That's why I support the idea of engineers taking support calls on rotation to understand the user.
Both can be true at the same time, a product manager who retains the big picture of the business and product, and engineers who understand tiny but important details of how the product is being used.
If there were indeed perfect product managers, there would no need for product support.
A lot of the error messages I'd write were for me, especially those errors I never expected to see.
The typical feedback I'd get from end users is "your software doesn't work". If they can send me a screenshot of the error I'm halfway to solving the problem.
Similarly, by siloing the world model in one or two heads, you disable the team dynamics from contributing to building a better solution: eg. a product manager/designer might think the right solution is an "offline mode" for a privacy need without communicating the need, the engineering might decide to build it with an eventual consistency model — sync-when-reconnected — as that might be easier in the incumbent architecture, and the whole privacy angle goes out the window. As with everything, assuming non-perfection from anyone leads to better outcomes.
Finally, many of the software engineers are the creative type who like solving customer problems in innovative ways, and taking it away in a very specialized org actually demotivates them. Many have worked in environments where this was not just accepted, but appreciated, and I've it seen it lead to better products built _faster_.
That's because it's much, MUCH faster to do it that way, though if you can deal with certain type of latency trade offs for throughput something like turbopuffer can do wonders for your costs.
> why would you not want to index?
Because if you don't need an index it wastes RAM, as you've learned. Maintaining indices also has a cost. Index only what you need.
In the sense of the blog post: A senior with decent DB experience would have told you. ;)
I am not experienced with MongoDB, I don't know if previous comment reports were the users fault or MongoDB's. But one thing is clear to me, complaining it uses too much RAM and not knowing the reasons for it, is a user problem. A common mistake is to setup a DB and expect it just magically does works. DBs are complicated beasts, you have to know how to deal with them.
In all fairness this was my first job a few years ago as a developer, I deep dove MongoDB but I was also one of the only devs using it at this place.
My previous experience with MongoDB had been in college and more limited.
At some point they added the docValues configuration option per-field to do the transformation during indexing and store it to disk instead, so none of it had to be stored in the heap. Instead what you're supposed to do is rely on the OS disk cache, which handles eviction automatically, so you can run with significantly less memory but get performance improvements by adding memory without having to change any configuration further.
So that we're on the same page, what I think should be PM responsibilities:
If I have a user story: "As a customer I want to purchase a product so that I can receive it at my address" - PM defines this user story as they have insight to decide if such feature is needed.
PM should then define acceptance criteria: "Given customer is logged in When they view Product page Then 'Add product to basket' button should appear", "Given 'Add product to basket' button When customers click on it Then Product information modal should appear" etc - PM should know what users actually want, ie whether modals should appears, or not; whether this feature should be available for logged users only, or not.
How this will work shouldn't matter to PM; these are AC they've defined.
Of course the process of defining AC should involve developers (and QA), because AC should be exhaustive to delivering given feature
In your example of an order placement - the PM has no special knowledge of what is a good customer order flow. Developers are usually way better at coming up with those by the dint of experience and technical knowledge of the current codebase and make the appropriate speed/polish trade-off.
PMs acts as an imperfect proxy for what the customer wants, making judgements off nothing more than their own taste. And though there are many great PMs, the taste of a PM is usually worse than that of developers and designers on average.
IMO the main business reason they exist is for organization accountability and ownership, despite the often negative value they bring.
List comprehensions are inverted (syntax-wise) compared to regular program flow, but that is pretty easy to learn and adapt to (and is, imo, much better than "a = b if x else c").
items = [ this for iterator ]
Again the bigger problem is the 'if', including in comprehensions.
I think these are realistic expectations for most apps. Obviously the likes of Netflix and Uber get orders of magnitude more, but 99.9% of apps aren't a Netflix or an Uber, and you don't have to optimize for scaling until your app is on a trajectory to become one, and putting your database on an SSD already let's you handle several thousand concurrent users with ease.
Of course everything depends on use case and constraints. I highlight the extremes here, the initial confusion was why DBs require so much RAM. Traditional DBs are optimized around RAM, that's where they perform best. You can abuse that, but it's not the best they can be in terms of latency, predictability and stability.
But that's simply not true at all.
Let's start with token by token:
items
items =
items = [
items = [ this
items = [ this for
items = [ this for iterator
items = [ this for iterator ]
Also `this` is a useless identifier, if you cannot transform or filter in your list comprehension. I don't like mine either that it contains pointless words...
Don't get me wrong, your example is clearly a right to left data flow. Which is not inherently bad, because `items` and `this` are new identifiers, which won't figure out by IDEs, so it doesn't matter.
Also, in my example of Python code (not my version of it, but the valid Python code), there is no need to have `if` at all to break intellisense, or break either left to right or right to left data flow several times inside the list comprehension.
Compare it to
family_adults = [ person for person in family_members if person.age >= 18 ]