I do a lot of support work for Control Systems. It isn't unheard to find a chunk of PLC code that treats some sort of physical equipment in a unique way that unintentionally creates problems. I like to parrot a line I heard elsewhere: "Every time Software is used to fix a [Electrical/Mechanical] problem, a Gremlin is born".

But often enough when I find a root cause of a bug, or some sort of programmed limitation, the client wants removed. I always refuse until I can find out why that code exists. Nobody puts code in there for no reason, so I need to know why we have a timer, or an override in the first place. Often the answer is the problem it was solving no longer exists, and that's excellent, but for all the times where that code was put there to prevent something from happening and the client had a bunch of staff turnover, the original purpose is lost. Without documentation telling me why it was done that way I'm very cautious to immediately undo someone else's work.

I suppose the other aspect is knowing that I trust my coworkers. They don't (typically) do something for no good reason. If it is in there, it is there for a purpose and I must trust my coworkers to have done their due diligence in the first place. If that trust breaks down then everything becomes more difficult to make decisions on.

The zinc gutter had leaked for probably decades and it destroyed part of the roof structure. The roof was held up by the wooden paneling the used to cover it on the inside (70s). So the wooden paneling was actually load bearing

Actually I've found way more stuff in this house. For example at the end of the roof the tiles weren't wide enough and instead of buying extra roofing tiles, they decided to fill it with cement and pieces of ceramic flower pots.

Software differs from all other means of production in that we can in fact test any change we make before realizing it in the world.

With good tests, I don't care what the intent was, or whether this feature has grown new uses or or new users. I "fix" it and run the tests, and they indicate whether the fix is good.

With good tests, there's no need for software archeology, the grizzled old veteran who knows every crack, the new wunderkind who can model complex systems in her brain, the comprehensive requirements documentation, or the tentative deploy systems that force user sub-populations to act as lab rats.

Indeed, with good tests, I could randomly change the system and stop when I get improvements (exactly how Google reports AI "developed" improvements to sorting).

And yet, test developers are paid half or less, test departments are relatively small, QA is put on a fixed and limited schedule, and no tech hero ever rose up through QA. Because it's derivative and reactive?

This is to say —- I agree with the article, but much nicer is to work at a place where you don’t expect to make this particular discovery very often, hah

The analogy for the load bearing stud might be a hackathon project that never expected to see production. In reality, a lot of what we do is hack on something until it barely works, and move on to the next thing.

> Except that over time it had become accidentally load bearing

> through other (ill conceived) changes to the structure this stud was now helping hold up the second floor of the house

Evidently, you couldn't easily tell why it was there. Moreover, I'm not persuaded that it accidentally became load bearing. It seems quite plausible that it deliberately became load bearing, for reasons which are ill conceived to you but not to the people who had them.

The lab was full of smart people who were used to looking at things and making their own well-reasoned conclusions about whether it was OK to change something. This was a warning not to be too hasty about doing that!

IME, having robust alerting and monitoring tools, good rollback plans and procedures/automation should eliminate this fear entirely. If I was afraid to touch anything for fear of breaking it, I'd likely never get anything done.

When I come in on a consulting basis I often have to help developers unwind the unintended effects of years of patches before we can safely make seemingly-simple changes. It's analysis-intensive, and like an archaeologist any artifacts I can dig up that you've left behind providing clues to what was in your head at the time can be helpful. In some cases where a set of functions is super critical we've made it part of the culture every time altered code is checked in to perform a walkthrough analysis to uncover any fresh unintended side-effects, ensure adherence to design requirements and discover and spotlight non-obvious relationships. The key is to turn the unintended or non-obvious into explicit. Sounds painful but in practice due to the high risk attached to getting that portion of code wrong the developers actually love that time for this is worked into the project schedules - it helps them build confidence the alteration is correct and safe.

I wish it were easier to impress the importance of this on inexperienced developers without them having to go through the hell of maintaining code in an environment where it was lacking.

It's a skill and art to keep the appropriate level of conciseness to avoid the documentation (no matter which form it takes) becoming too verbose or unwieldy to be helpful.

We're probably at a stage where doing this in an automated fashion might be reasonable. The context for why something works a particular way (say in some github repo) is the commit message and change detail, the PR, related issues, related chats in <chat tool of choice>, and then things temporally related to those. Feeding an LLM with the context and generating ADRs for an entire project automatically could be a feasible approach.

Dependencies flow in and onto new components like water. This is a fundamental law.

Even secret interfaces, etc that you think no one could find or touch will be found and fondled and depended upon.

This is why the correct view is that all software is a liability and we should write as little of it as possible, just the minimal that is needed to get the dollars and nothing more.

A good example of a famous failure if this:

https://en.m.wikipedia.org/wiki/Hyatt_Regency_walkway_collap...

That's why it is a good practice to have DiRT or similar exercises. Break things on purpose so that people don't rely on them when they shouldn't.

One of the biggest challenges I find in refactoring/rearchitecting is getting people to separate why something was done from whether it still needs to exist.

Too many times someone has insisted that a weird piece of code needs to exist because we never would have shipped without it. They treat every piece of the structure as intrinsic, and can't conceive of any part of it being scaffolding, which can now be removed to replace with something more elegant and just as functional.

When you put a doorway through a loadbearing wall, contractors and smart people place a temporary jack to brace the wall until a lintel can be substituted for the missing vertical members. Some developers behave as if the jack is now a permanent fixture, and I have many theories why but don't know which ones are the real reasons.

On the other hand, if you can't figure out what something is for, sometimes the easiest way to find out is to remove it and see what breaks. Maybe that's not such a great idea for that load-bearing stud, but in software it's easy to undo.

Guess I'm thinking of being very aware of how the structure reacts to failure, and not necessarily the strenght of all individual parts.

If you ever see a house stripped down to the sticks for a rebuild, you will hopefully notice a few braces added. Not to keep the walls from falling down, but to keep them square and true until the walls are rebuilt.

At first look it seem like someone backed into the garage door and mangled the hell out of it but on more careful inspection the roof is being barely held up by the tracks that the door runs in and is pretty near to giving up the ghost. Was just going to splice the ends of the rafters (like someone did on the other side who knows how many years ago...if it works, it works) and replace the garage door but now its looking like I'll need a whole new roof.

What really worries me is the dodgy wiring strung all across the basement which is a combination of newish wires, old cloth covered wires and liberal applications of electrical tape to splice it all together. Luckily none of the wires seem to be load bearing...

Except when the tests verify what the code was designed to do, but other systems have grown dependencies on what the code actually does.

Or when you're removing unused code and its associated tests, but it turns out the code is still used.

Or when your change fails tests, but only because the tests are brittle so you fix the test to match the new situation. Except it turns out something had grown a dependency on the old behavior.

Tests are great, and for sufficiently self contained systems they can be all you need. In larger systems, though, sometimes you also need telemetry and/or staged rollouts.

Your code was supposed to calculate the VAT for a list of purchases, but progressively becomes a way to also force update the VAT cache for each product category and will be called in originally unexpected contexts.

BTW this is the same for comments: they'll cover the original intent and side effects, but not what the method/class is used for or actually does way down the line. In a perfect world these comments are updated as the world around them changes, but in practice that's seldom done except if the internal code changes as well.

Having worked on very long-running projects, testing or well-funded QA doesn’t tend to save you from organizational issues.

What typically seems to happen is tests rot, as tests often seem to have a shelf life. Eventually some subset of the tests start to die - usually because of some combination of dependency issues, API expectations changes, security updates, account and credential expirations, machine endpoint and state updates, and so on - and because the results from the test no longer indicate correctness of the program, and the marginal business value of fixing one individual broken test is typically very low, they often either get shut off entirely, or are “forced to pass” even if they ought to be an error.

Repeat for a decade or two and there quickly start being “the tests we actually trust”, and “the tests we’re too busy to actually fix or clean up.”

Which tests are the good ones and bad ones quickly become tribal knowledge that gets lost with job and role changes, and at some point, the mass of “tests that are lying that they’re working” and “tests we no longer care to figure out if they’re telling the truth that they’re failing” itself becomes accidentally load-bearing.

Huh? Your first part seemed to be repeating TDD optimism but then you switch test departments. Just make your claims consistent, I'd suggest you instead talk about tests being written by the programmers, kept with the code and automatically run with the build process.

However, I don't think even TDD done right can replace good design and good practices. Notably, even very simple specifications can't be replaced by tests; if f(S) just specified to spit out a string concatenated with itself, there's not obvious test treating f as a black box that verifies that f is correct. Formal specifications matter, etc. You can spot check this but if the situation is one magic wrong value screws you in some way, then your test won't show this.

there's no need for software archeology, the grizzled old veteran who knows every crack, the new wunderkind who can model complex systems in her brain, the comprehensive requirements documentation, or the tentative deploy systems that force user sub-populations to act as lab rats.

Wow, sure, all that stuff can be parodied but it's all a response to software being hard. And software is hard, sorry.

Software needs people with the suspicious minds of good testers, but security people make more money for the same skillset.

"You" might know it in "your" creations, but in my career I am much more often working and reworking in other people's creations.

I think the point of the article is not that you should avoid using decorative studs as load-bearing elements, but that you should be aware that others may have done so before you came along.

This is an even more conservative position than the default Chesterton's Fence reading, which is itself dismissed by a lot of people as pedantically restrictive.

For me, the parent article resonates. I have definitely had ceilings come crashing down on my head when I removed a piece of "ornamental" trim (programmatically speaking)

Is it always lazy in the bad way, though? In software there's no sharp distinction between "built to carry weight" and "built to tack drywall onto." Whether a system is robust or dangerously unscalable depends on the context. You can always do the thought experiment, "What if our sales team doubled in size and then sold and onboarded customers as fast as they could until we had 100% of the market," and maybe using your database as a message queue is fine for that.

If it results in a sad dev team, then that's a case where it was a mistake. It's hard to maintain, or it's an operational nightmare. That isn't the inevitable result of using a (software) decorative stud as a (software) load-bearing element, though. There are a lot of systems happily and invisibly doing jobs they weren't designed for, saving months of work building a "proper" solution.

More often than not I've seen this happen because they, in fact, do not know.

You could conclude that the code is unnecessary and remove it, or you could conclude that some test cases need to be added to exercise it. How do you decide which is correct?

The problem is usually that well thought and and designed software was build for a moving target, and invariably things have changed over time. It's not necessarily a sign of lazy design, it's where the real world intersects with the nice neat pretend world we design for :)

Do you? Sometimes quick-one-time fixes becomes the center of important software.

Whenever I find myself doing this, I at least leave a comment typically worded along the lines of "the dev is too lazy, not enough time to do it right, or just has no clue what to do, so here you go..."

Hidden files in unix were a bug.

Lots of EE’s need to do some programming, and lots of people getting EE degrees end up in programming careers, so it would be a disservice not to teach them any programming at all. In particular, an engineer should be taught enough programming to put together a matlab or Numpy program, right?

Meanwhile, some of their classmates will go on to program microcontrollers as their job.

Writing programs and a product, and writing programs to help design a product, are two basically different types of jobs that happen to use similar tools, so it isn’t that surprising that people mix them up.

I think part of the thing is Software Engineers haven't been a thing for as long in the industry. I'm the only Software Engineer I've met doing controls. My supervisor has a CS degree and an Electrical Technician diploma, but never another SE.

Second is I think up until recently, the work done by Control Systems has been what's capable of an EE or ME so having a SE hasn't been necessary. I've been with my company for 10 years now, and in that time I've watched the evolution of what my clients are seeking in terms of requirements to their systems.

I primarily work in Agriculture or Food Production. 10 years ago my projects were assembling plants and getting their motors to start, with the required protections then some rudimentary automation to align paths and ensure flow.

Today? I'm building traceability systems to report on exactly which truck load was the origin of contamination for a railcar shipped months later. Or integrating production data to ERP systems. Adding MES capabilities to track downtimes and optimize production usage. Generating tools to do root cause analysis on events... It's a different world and the skills of a Software Engineer haven't really been a super important role for quite a while.

I think the mindset is shifting, but it is slow.