Technical Debt: How do you get out of the bottleneck?(martinfowler.com) |
Technical Debt: How do you get out of the bottleneck?(martinfowler.com) |
Sometimes I wish these articles would just state: Stop sprinting
Also: document your decisions! If you have to write down and justify your (prudent) technical dept, you maybe catch some stuff before it's written in code. It also makes sure more people are aware of it. Additionally, it sometimes prevents CV driven development, which imho is a big problem in tech. Overall, documentation is a very underused tool.
Finally: teach everyone (especially marketing) that software development is expensive, so everybody thinks twice before wishing for nice to have features done quick.
Steve Jobs' presentation of Snow Leopard should be mandatory viewing for every manager. "No new features" to a standing ovation. Customers actually like stable, responsive, and efficient tools, who would have thought?
EDIT:
> Bringing "microservices" into the room is definitely not helpful
It's a pretty good way to increase technical debt. The microservices themselves might become small enough that it is easy to refactor some at a time, but then you have the actual deployment and communication infrastructure to contend with. And if you get that architecture wrong, oh boy you'll be begging to go back to a monolith in a day.
I remember at a previous job, the monolith was drowning in technical debt. Someone decided the solution was Go microservices. Fast-forward 18 months and 95% of the functionality is still in the monolith, but there are now 25 microservices, and no environment (except production) where you can test everything together.
Additionally to managers watching that presentation, software architects should mandatory have to work as normal developers in every project they designed for a a few months. Best, if it would be around 2 years after they came up with the architecture. If you don't taste what you cook, how can you learn?
I fully agree on your points with microservices.
Stop sprinting is good but also : stop following Google. You are not Google. You do not have the needs of Google. Just stop it.
Pick the simplest tech stack you can find. This gives 100x more time to your developers to work on company solutions and not fighting technical complexity.
_Existing_ customers like these things (and they aren't wrong for liking these things).
New business comes from the potential customers who weren't enticed by the existing feature set, robust or otherwise. You bring them in by adding new features. The retain/new business priority is often heavily weighted in favor of the latter.
Retaining existing customers stuck with a crappy product ain't that hard between sunk cost/lock-in effects and the evergreen insulation of people with purchasing authority from the day-to-day pain inflicted by their purchasing decisions. A couple fancy steak dinners for middle management effectively papers over the cost of driving a department of ops engineers to cirrhosis several times over.
I've been through this slog - sometimes it requires fixing what's there and other times, a rewrite. People hate rewrites, but it really is the best way forward in a lot of cases if done correctly.
This is my experience with many different systems. I’ve found that mentoring and personal development identification techniques for what to learn, and when, can be helpful. Inviting a developer to be part of the decisions made regarding their future may seem obvious, but in practice it’s often not done…
And documentation… I’m known as “Just write it down,” but there is usually significant pushback… from all parties.
This is Fowler (and Uncle Bob's) stock in trade tbh.
Generic good advice that sounds plausible in a sane world.
Without ever acknowledging that we actually don't live in that world.
Getting it right the first time is a very hard game to win. It’s best to save some of your energy for the times that really count. It’s always interesting to me when coworkers exclaim that doing the right thing is too hard. Reminds me of myself at age nine trying to get out of chores. More seriously though, “if it hurts stop doing it” is how dumb animals think. Pain is information. Ignoring it is dumb. Almost as dumb as giving up is.
Just last week I was having a hell of a time getting some code to work. Running into a wall. Okay fine, I’ll write more tests. Still struggling. Oh hey, you know what would make this way easier? If I rearranged this code in the manner I thought about this morning but decided to not work on until tomorrow.
If eating the code is difficult, there’s a point of very quickly diminishing returns where adding more logic to the tests is making things worse, and you should think about whether The code is too complicated to test. Maybe you need to remove code, instead of adding it twice.
Clearing debt can be a bit of a fishing expedition, but as long as enough of them have a payoff, it hardly matters if occasionally they come up goose eggs.
I've tried to articulate this (a bit tongue-in-cheek) here: https://www.evalapply.org/posts/software-debt/
In short, I feel a re-scoping in terms of "Software Debt" is warranted. And a re-casting of the risk of this debt in terms of the rocket equation, and opaque financial derivatives of the kind made infamous in 2007/08.
But it only ever happens when the hands-on people get to spend significant time on improving what is happening behind the scenes. That might mean 20% or 50% of their time, not the 2% or 5% that many get.
If the hands-off people calling the shots aren't willing to make that commitment then a heavily indebted organisation's fate is already sealed.
Shameless plug: https://leadership.garden/tips-on-prioritizing-tech-debt/
There’s sometimes bad code that one resolves by redesigning some part of it with eventually some glue code with the old system, documenting it and making a presentation for everyone on how should the new stuff be written, and distribute work for a slow migration of the old systems by the different parties.
I’ve seen also some systems designed to solve problem X while the company or industry moved on to problem Y. Again, act like a virus, infect the system with your brand new solution, glue it or bind somehow with the old system and let it spread.
The only mistake that you can do is to engage all of your resources to undertake a full redesign and rewrite everything from scratch.
And for processes or algorithms that don’t scale, it’s not a debt because you never really made the investment.
From [1]:
> In my Ruby code, half of my methods are just one or two lines long. 93% are under 10.
Are there professional engineers out there who reads this tweet and thinks striving for one or two lines methods is a good idea? I really don't understand how does this person have so much fame or so much authority on architectures? What exactly are his accomplishments other than selling books and seminars? Can someone please explain how come Martin Fowler is so revered among developers?
[1] https://twitter.com/martinfowler/status/893100444507144192
Striving for the shortest possible methods (less than 10 lines is good) is a standard in Smalltalk. It's not a bad idea, at least, it's been shown to work over and over again over the last 50 years. Ruby takes a lot of inspiration from Smalltalk, although it lacks the biggest factor that made short methods work so well in Smalltalk: the interactive editing of the code as it runs.
Another example where a subroutine longer than a few lines is frowned upon is Forth. Also in that case, interactive editing is the main motivation, along with the fact that the amount of irrelevant bookkeeping grows linearly (or worse) with the subroutine length (unless you use local variables or the return stack, which are both hacks in Forth).
In any case, "professional engineers" who don't even know Smalltalk... No, I mean, the history and breadth of their field, probably should not be too judgemental about what works where, outside of their immediate expertise.
1-line methods is taking things to an extreme (though you can write a lot of logic into a single line in Ruby, so maybe it's not as crazy as it would be in C...) but I found the discussion in his refactoring book about reorganizing methods and naming things very illuminating.
It's an idea so simple it sounds stupid sometimes to try to explain to people, and yet, I never stop seeing new code that could be improved by something as simple as methods with accurate names.
E.g. I've seen a lot of stuff like:
```
// This does X
[some obtuse nasty inline regex or five-layers-deep nested object call or otherwise crazy individual line of code]
```
That maybe eventually gets detached from that comment line and now there's just some nasty line that nobody on the team understands anymore that can't as-is be easily tested in isolation from a bunch of other stuff.
A perfect opportunity for a one-line method, that can have a descriptive name and can have its own specific tests.
And yeah, again, that's the extreme case, but most codebases have tons of opportunities for 5-10 line methods being extracted with helpful names. If you find yourself writing a comment to describe a block of code, maybe make that comment your method name instead. And sometimes you try it, and realize "wow, it's hard to extract these methods without an insane amount of input params and return values for each bit" and maybe that's an important thing to realize about the code you're looking at anyway. ;)
M. Fowler, Uncle Bob, Kent Beck, GoF authors...
Any book or post by these people is always brought up as a must read. A must read to understand why you should *not follow cargo cults* and see with your own eyes how toxic and damaging these guys are to software engineering culture.
In one hand, I'd say these guys are absolutely revered among computer science students (their main target to sell their books) and developers who have been stuck in the OOP and design pattern rabbit hole for decades without evolving.
In the other hand, the criticism is more present that ever. It's ironic, because the criticism has even lead Uncle Bob to jump the wagon and start to sell Functional Programming in Clojure as a new super *secret formula* and *best practice for real engineers*.
Organizing all those small functions can be a challenge, though. Which is why I don't tend to have functions quite that small. But a lot of 10ish line functions tends to work ok. But then, Erlang doesn't have loops, and I don't like anonymous functions (unless they're so short and simple I don't expect to find them in a stack trace), so that tends to make things short by necessity.
I'm somewhat glad I made it out the other side; as a babe in arms programmer I was fully taken in by these ideas. It's a great game of fun creating these overly complex, but ultimately arbitrary, sets of rules. They serve no real purpose other than creating and solving mental puzzles. There's a reason Software Architects don't (shouldn't) exist any more.
I have the opposite anecdote. Currently working on a monolith that's been around for 5-7 years, huge enterprise Java mess, like many others in the industry. The clients decided that they'd like to upgrade to JDK 11, newer frameworks, all of that shiny stuff.
So for the past 4 months i've essentially been pulling my hair out and trying to rewrite significant parts of it all. When you have a monolith, you cannot upgrade the entire system if some parts of it break - even if i have, say, 200 dependencies but 10 break, i cannot move forwards with the updates and as a consequence am stuck with running on JDK 8 or even outdated frameworks.
And, of course, you cannot extract those parts of the system out either because you'll immediately be hounded by developers who aren't welcoming of change and will find nitpicky stuff to tear both your arguments and efforts apart, actively sabotaging any potential successful outcome (potentially exaggerating here, but many do not enjoy change).
Alas, there is probably some sweet spot to work towards from day 1. Not going crazy with microservices, especially due to how people interpret the "micro" part (e.g. service per person vs service per team, what the total count should be, domain modeling etc.), but not sticking to a single large monolith either.
I think that sooner or later the industry will try grouping code into services based on the "type" of functionality - the weird PDF export and reporting logic will live in service A, other attachment upload/download logic in service B, the web API in service C, and the old legacy server side rendered UI in service D. That way, at least your efforts to update the web framework and JDK for it wouldn't be usurped by the PDF library not liking it.
Then again, i've seen front end applications baked into back end applications instead of separate back end/front end deployments far too often, so i'm not hopeful about anyone genuinely exploring that approach anytime soon.
Alternatively, i've actually written about modular monoliths before, in my article "Moduliths: because we need to scale, but we also cannot afford microservices": https://blog.kronis.dev/articles/modulith-because-we-need-to...
But it wasn't broken up because it was too hard to update dependencies, it was broken up because it was a complicated mess. They just traded it off for another complicated mess.
"Getting it right the first time is a very hard game to win."
I cannot disagree if it is one's first time yet this is my sixth and I have no losses to date. I have nothing but time and so in that time I will see how this game plays out.
Everything else is just annoyance, and lumping those in with the others ends up pulling attention away from the ones that really matter, meaning those solutions get compromised on quality.
Leaving space for a solution is not the same thing as creating an abstraction for a solution, and most people don't get that. Architectural astronauts elevate this blind spot to performance art, and none more profoundly than J2EE architects, as lampooned by Enterprise FizzBuzz: https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpris...
I wonder sometimes if some of my former coworkers' cognitive dissonance ends at the office door, or if at home they hang empty picture frames on their walls as placeholders for when they eventually find some art they like, put armchair-shaped boxes labeled "Chair" in their living rooms, dolls in empty cribs and stuffed animals in empty pet beds.
> [some obtuse nasty inline regex or five-layers-deep nested object call or otherwise crazy individual line of code]
… which no longer does X
The one that snuck up on me was sort of the corollary to Screechingly Obvious Code, and a cousin to Pandora's Box - which is, strictly speaking, and antipattern (except that in the end of the original story, Hope was left in the box).
If you have a function that is Obvious except for one small part, factor it out into its own function. The care comes in picking the incision points. The inputs and outputs to the new method need to be obvious (ie, don't factor out a function with side effects, that makes for spooky action at a distance), which may require some temporary data structures in order to iron out.
Going further, if you are considering a behavioral change that may not be obvious (performance optimization, error handling for very obscure cases) that has no prayer of being screechingly obvious, determine what lines will need to be changed, extract method on that code first, commit it, and only then make the change to the extracted code. Mikado method often comes into play here. Get used to revisionist commit history techniques.
What I've found through long observation is that people are more comfortable reverting such changes if they break when you are not there, which in turns make them more open to approving the change in the first place. "Will this bite me on the ass later?" is a pretty important consideration for most people. The commit history is very clean, and someone can revisit your decision years from now, when your user base and hardware are orders of magnitude bigger, when the runtime and hardware have very different timing ratios between different activities, and so on. And critically (for the optimization obsessed), compilers are very good at inlining leaf functions, which means the 'cost' of the context switch can be negligible in practice.
Also I don't know about anyone else but I've found that my attachment to such code is also fairly low. You are much more likely to get encouragement to 'go for it' when broaching the subject of undoing one of these changes, instead of a lecture about watching for regressions (I tend to "over-invest" in robustness, so deleting my code statistically represents a reversion to the mean wrt to code quality).
And perhaps more importantly to this conversation, they tend to tune out these leaf node methods when tracing code looking for bugs or trying to add new features, bugfixes. These structures don't nerdsnipe us, causing us to dump state on short term memory and have to start over. If I had a dollar for every time I said, "Aha! I found the place where we return null! Wait, why was I looking for that?" I could retire. This is, in my mind, the critical quality of Screechingly Obvious Code - in uses up a single slot of short term memory, facilitating code traversal.
There is a lot of merit in that argument.
It never ceases to amaze me how much effort we now spend on nothing but infrastructure and architecture. It's like the Cloud Way of "cattle not pets" and "infinite horizontal scalability" may no longer be questioned.
Meanwhile you could build almost anything in this space with little more than a rack of real servers, some respectable application code written in a fast language, and a bit of scripting and OSS for the glue.
Obviously that sort of environment wouldn't be sufficient for every modern, online-first application but how many years would it take to outgrow it? Many applications never reach that scale. The lucky few would be in a great position to expand from if their starting point was simple and transparent with minimal dependencies.
For all the talk about "no silver bullet", there's a staggering amount of folks who keep falling for it.
The idea that a shop that built a messy system will suddenly build a non-messy system just if they pick up a new set of tools - that they don't have practice with anyway! - is exactly the silver-bullet fallacy. The system turned out messy because it's complicated, switching to a very complicated tool doesn't take away the complexity, it just changes the way you manage it.
The last couple Kubernetes shops I worked at were that exact sort of hodge podge of custom glue with high tech debt and low bus factor. Just even doing something "simple" like "ssh into the box and poke around" when the two wizards are out suddenly has a huge learning curve. ;) So then you get into discussions of "oh you should've been using managed kube" or "oh, don't use TF -> helm -> kube, use [other way] of managing it" or blah blah.
After you fuck it up a few time you might have enough useful experience so that you can build it well next time... just pray that by then we're still using the same tools. :D