If you spend all your time refactoring, cleaning up after legacy design constraints, fixing ossified errors, then you run out of time and fail to write actual meaningful income generating features. Conversely if you make none of those improvements, eventually the weight of bad architecture slows all progress to a halt and no new income generating features get delivered.

One of the hardest parts about advancing as a developer, in my opinion, is being able to tell when you should refactor versus just leaving the old working mess alone.

With your example, it's like if it turns out that the dining table is embedded into the ground with concrete because it kept blowing over, and moving it indoors would require getting a carpenter to create new legs. And also that because the dining table has been there for so long, someone decided to run electricity cables through it, so rerouting it requires an electrician and will shut down the factory at the bottom of the garden for half a day. We could buy a separate table for indoors, to try and slowly migrate to the new table, but then we'd have two tables to maintain and we all know how that usually goes.

At a certain point, you look at it and go well the commercial dishwasher is just $9k and we can focus our efforts on building that loft conversion for now.

I have, too. And then I usually haven't managed to put the dining table indoors. And then new people came in and asked the same question you ask, and by then I was one of the people who tried to put the dining table indoors, and explained how it wouldn't fit through the front door, and how I tried to get it in through the window. And then the new people try to put the table indoors and fail and next thing you see they're either leaving the house or explaining to the newcomers why the table is outdoors.

Ultimately, I've realized that talk like this is cheap, unless you can actually improve things. That requires leadership skills and some political capital in your organization. I don't think the author of the article deserves an engineering leadership role simply for complaining about things. (They might still deserve an engineering leadership role for other reasons, what do I know...)

"The Boeing 737 MAX: Lessons for Engineering Ethics (2020)"

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7351545/

It's an example of a workaround that should not have been tolerated:

> "The Maneuvering Characteristics Augmentation System (MCAS) software was intended to compensate for changes in the size and placement of the engines on the MAX as compared to prior versions of the 737."

Rather shockingly this wasn't even an engineering problem workaround; it does seem that it was solely designed to avoid an aeronautical reclassification of the aircraft that would have required pilots to undergo an expensive retraining program on flight simulators, which might have caused lost orders.

This does look like a systems-level failure, but one at an organizatonal level: the system went from a state where engineering took priority, to a state where financialization took priority. In systems thinking, this could be called a state transition: a fluctuation takes place, and afterwards the system settles down to a new (apparently) stable state quite different from the old state:

> "One factor in Boeing’s apparent reluctance to heed such warnings may be attributed to the seeming transformation of the company’s engineering and safety culture over time to a finance orientation beginning with Boeing’s merger with McDonnell–Douglas in 1997 (Tkacik 2019; Useem 2019). Critical changes after the merger included replacing many in Boeing’s top management, historically engineers, with business executives from McDonnell–Douglas and moving the corporate headquarters to Chicago, while leaving the engineering staff in Seattle (Useem 2019). According to Tkacik (2019), the new management even went so far as “maligning and marginalizing engineers as a class”."

What’s even more infuriating is seeing new engineers join that team, question why the hell something insane is insane, and then slowly grow used to the insane thing. Only for the cycle to repeat when the next new person joins.

I make a huge point of confirming to them, that yes we do acknowledge it’s insane. Even if management doesn’t care.

Oh wow, that hits home. To be fair, the historical context for the decision can be valuable information, the problem is the next step. Even if you can't fix it right now, you might make steps towards that. Or you might say: Now that we have this heavy table outside, why not attach more things to it?

obviously this is more or less possible given specific organizations and personalities at said organizations. my point is more that the magnitude of the task of migrating the ossified organization to a better architecture, even with a fully pliant staff totally on board with changing, should not be underestimated.

If it were up to hindsight me, we'd have carefully designed and orchestrated every past decision. We'd do full design qualification and change control on projects and purchases, researching carefully to never make a mistake. We'd sit as a team and brainstorm every possible implication. We'd write, execute and document tests for everything to prove ourselves. Any sniff of inefficiency and we would stop everything and fix it, no matter the cost. We'd take time to document, investigate and follow through. It would be a glorious cavalcade of plans and CAPAs, qualifications and tests. And reports! Binders and binders of wonderful validation reports everywhere!

If it were entirely up to hindsight me, we'd run our little widget company like we're building a space shuttle. Of course, we'd never make any money. But we'd be doing it right, by gum!

Most companies need to be somewhere in the middle. No hindsight and you end up a tangled mess of short-sighted kludges, all hindsight and you can't move forward. Either way you risk ending up a lead balloon.

If you find yourself in the kludge company territory, then here's some advice from a talk I recently attended[1]:

Start by training yourself and your team in Root Cause Analysis. Empower them to start thinking deeply and critically about what's really causing the problems and inefficiencies you encounter. Understanding root causes naturally translate into solutions that aren't just bandaids. Use these skills in your day to day, and you'll start building a culture of quality around your systems.

[1] Steve Gompertz

In my experience, at the time the decision was made, folks did want it that way. The organization has lost that context as to why and has only documented the technical design.

A curse shared by less effective engineers I've worked with is to rage at legacy decisions unable to convince the organization to revise them. They lack the ability to understand the various stakeholders involved and to come up with a plausible plan. A systems engineer (as referenced in the blog) would understand the various sub-systems that make up an organization and be able to drive the change they desired (the conclusion that it's irreparably broken or you lack the expertise to fix it would be fine too).

Love your post, though!

I'm reminded of back when I was studying pattern recognition for a system that would become an Expert System (this was before that term was used). I would read many articles saying what techniques would work. I had the urge to ask "But this doesn't show how you got there. Show me your discarded solutions that didn't work." I would like to see your wastebasket.

Similarly, I am inclined to someone who acts as an expert to tell me five ways that won't work.

The only logical reason to do this is because it has no impact on the business. Or at least, smaller impact than a total rewrite/refactor would have.

If an engineer presented a case where fixing an underlying issue resulted in better business outcomes vs. a short term band-aid, then I don’t know anyone that would tell them no. Businesses want to succeed. They want to make money. If you can help me make more money I’ll let you do whatever you want (within the confines of the law and civil society).

"What you gotta understand is that in the beggining of Electronics, people were pretty much trying to put two materials together they thought worked and then tried modelling it. It was pretty much trial and error, experimentation..." and then he hits me with the most "holy s*" moment of my academic life: "... much like programming and software engineering is today. You write some code, run it, see if it works. Works, ok, go ahead, make sense of it, explain in the documentation, next task".

I had NEVER thought of software like this, it just hit me like an atomic bomb in the head, I felt like I understood where in the history of software engineering we are right now. Structure is coming, slowly but surely.

I initially watched the lectures because of an interest in aerospace and it is a fascinating historical series in its own right with some incredible speakers. The lessons for systems engineers are numerous too.

I'm not sure if I understood it correctly or there was a typo, I am surprised because my experience is almost opposite.

my experience with Swedish and German multinationals is that it's all about consensus, to a point where the best decisions would be rejected if it lacks support, or where the off-ramp would be no decision (which is also a decision). this I found in stark contrast to French, Italian, Spanish, UK/US/Australian organizations, that tend to value the ego driven hero who saves the day. Also these former locations seem to do better when dealing with chaos as they know how to "think on their feed".

I'm curious from your experience would these companies be start-ups or medium sized firms, or could there be other reasons I'm missing that our experience is so different?

I don't think, that this is necessarily bad. I even think, that this is a must for a large team. But it changes the way you code.

Right now I have the time and opportunity to 'form' the code the way I want. I can try different approaches, rip out or rewrite bad code and add features as I see fit. That's what I call liberty. I don't feel like a raw coder. Instead I feel like an artist in my own universe I where I'm in charge.

This feeling can get lost, when you need to request, plan, control and review every single change in coordination with multiple team members. You're not in charge anymore. You're not the artist, but a cog in the wheel.

I think some people like one way or the other. But it's hard to transition.

Here is the hidden truth. So much of the current information sector is just daycare for grownups. Then there's a secret nucleus of people who actually do the real work. The secret to happy employment rests on being able to determine who's actually cutting lumber versus who's just playing dress-up.

If we want to gradually move towards a system of universal basic income, maybe we could help sell it by funneling larger sections of society into IT, and just give them a bullshit job where they can fingerpaint all day to get their paycheck. Eventually you can let them stop fingerpainting and just give them the paycheck.

> The factory of the future will have only two employees, a man and a dog. The man will be there to feed the dog. The dog will be there to keep the man from touching the equipment.

My version of this is that those handful of people become critical to the organization, which must be avoided at all costs. Every place I've been that's weighed down by huge IT staffs, undervalues tech, absolutely hates to give coders raises, and sees tech as a cost center, even while preaching how important technology is.

Sadly, this seems to be the case so often that the only companies that really succeed with good tech have builders in the founder's chair who _still code_, at least until the model is firmly established (but even after is good too). Pretty much everybody else resents software devs and thinks they're overpaid.

Weirdly a smaller, better paid team can often be resented _more_ (because of course, you can't see all the money that's being _saved_ by having a small highly-powered team).

This is only true if the complexity comes from the underlying problem that must be solved or goal that must be achieved.

Complexity that exists for historical reasons, derived from technology or platform choices, solves a problem you don’t really have or don’t need to have, relates to the structure of the organisation, etc. etc. can of course all be made to go away completely.

In my experience a small fraction of the complexity in most organisations and software is truly, fundamentally unavoidable.

There are always processes. Sometimes they are explicit and well-understood; sometimes they are hidden, and not open to improvement because nobody knows what they are. Implicit processes tend to disadvantage newcomers, as well as reinforcing social assumptions and conventions (that tend to disadvantage those that are already disadvantaged).

Explicit processes don't have to be heavyweight and bureaucratic.

There was a good essay on the problems caused by informal/implicit processes, from about 20 years ago. I've spent 45 minutes searching for it, but I'm afraid I can't find it.

Absolutely. All too common. And once you do include them each is not merely a new variable capable of assuming wildly different values, but a whole system in itself capable of interacting with every other such system within your system. That's why reductionists like to try factoring them out as interchangeable cogs. Pretty much the entire edifice of modern industrial economics since Adam Smith and Henry Ford is built on that model simplification/efficiency.