Ask HN: Do you have real-world experience of technical debt killing a project?

3 points by hugorodgerbrown 11 years ago | 6 comments

Explaining the impact of technical debt to someone non-technical who hasn't experienced it first hand is almost impossible (e.g. "why didn't you just do it properly the first time?").

It would be great to get a thread here detailing real-world examples of debt killing a project, or more, that we can all point people to when they claim it's just developers complaining.

Sample experiences worth sharing include:

* Have you been part of a project that ground to a halt because of technical debt?

* Have you ever had to abandon a project/product and "start again from scratch" because of problems in the previous codebase.

* Have you ever left a job because the overhead of servicing technical debt made the job impossible?

* Have you ever worked for a company that went under because debt was holding them back?

I know it may not be possible, but real company names would definitely add value.

jdubya 11 years ago |

I worked for one of the largest load balancer/application delivery controller company for a grinder of a boss on a security team there.

My job was to hack shit, find vulnerabilities and write automation for attacks that our systems could prevent. There were 4 pretty big code bases (>100k lines) that were used for the same thing. There was no inter-team effort, everyone wanted to roll their own rather than working together because the bosses had huge egos and wanted their name in the brick.

I was tasked with building a new framework!

So yeah, five different automation frameworks, one for automating attacks against devices and servers, one for smoke testing (pound the box with traffic) etc. Mind you all of this shit could be done with a single framework because many of the tasks were similar. So I created a way to build test profiles from our test plans and consolidated the code bases. I learned a shitload about dealing with existing code during this effort. It was awesome in that respect, I learned how to deal with poor management decisions - execute better than they ever did.

Long story short I refused to scrap everyone elses hard work even though my boss told me to "build it from scratch and use it for the security team only". The reason being is because starting from scratch would have incurred more technical debt than the consolidation effort took. Mind you every other department in eng could have used said framework. Anyhow, under the radar I went about consolidating the other four frameworks and I gave my boss the timeline without his knowledge of what I was doing.

Three months later every eng department was using this new framework using the tools that they were used to using for deployment and verification, just a new ui and a lot of code that was cleaned up and everyone was pretty happy with it to be honest. Everyone that actually used it that is.

My boss, who never used any of the frameworks, got pissed off that I did not build a sec. team only framework and asked me to build a version only for the security team for "security reasons". This was the proverbial straw that broke the camels back. This fucking moron wanted to satiate his own ego at the expense of my peers time, health and the company. I don't really care too much for the company but everyone I worked with was very competent and I did not want to just throw the baby out with the bathwater. That would be stupid.

Long story short, I was interviewing at places at the time and got an offer that very day so I decided to take the offer.

Moral of the story:

Don't have your engineers build 4 different automation frameworks and then ask an engineer to "rebuild" a custom version for a bullshit reason. Don't build shit from scratch unless you are learning/experimenting and at large companies, they rarely reward experimentation.

I get that this might not exactly be technical debt but 4 frameworks and the effort I had to put in to the consolidation was brutal at times. Improperly used technology, poorly deployed interface supporting services and solutions for transient failures of tests being "just reboot the test box" was too much bullshit for me.

God I fucking hated that job. I will never work for a large, well established company again if I can avoid it. My startup might be getting picked up soon by a conglomerate and I have 6 months left on my vest. If we end up at the conglomerate I am going to quit and take a few years off working.

That is my experience.

wikwocket 11 years ago |

The best way to explain technical debt is by analogy: Technical debt is what happens to your entertainment center when you buy a new Playstation. To do things properly, you should buy another HDMI cable, run the cable through your cabinet around to the TV, on a dedicated input.

But that's not how it happens. When you bought the Playstation you also got a copy of Call of Battlfield Honor Brothers, and your friends aren't going to wait half an hour while you do that. So you steal the AV cords from the Wii, run them in front of the DVD player, and plug them into the XBox's TV port.

So now, the system still works, but it's not ideal. You have to move some cords if you want to play a DVD. You have to unplug the Playstation if you want to play Minecraft on XBox. And heaven help you if you want to play Wii Bowling.

Now imagine this happens many times over several years. You get a new XBox. And a Bluray player. And a new sound system. Each time you do what it takes to get things working, and no more. Pretty soon, just to watch Netflix you have to rewire a whole switchboard of cables.

At some point, you need to pay off your technical debt. In this case, by taking a whole Saturday to unplug everything, buy some new cables, and re-route all the connections in a clean, logical way. It's the last thing you want to do while your friends are playing Band of Duty-bound Heroes IV, but if you don't, you'll be rewiring Ethernet cables in order to watch Hulu until the end of time.

hugorodgerbrown 11 years ago | |

Thanks for the analogy, but this is missing the point - this thread is about the opposite - real, verifiable, experience.

wikwocket 11 years ago | | |

No worries. When you said "Explaining the impact of technical debt ... is almost impossible," I thought you (or others) might be interested in a good way to explain it. This analogy has helped me, so I hope others may find it useful.

G_rupture 11 years ago |

I haven't heard of any company going down because of technical debt, but here is a study that has some quantitative analysis of technical debt in the beginning: https://medium.com/building-bowery/e5a006f7b724

hugorodgerbrown 11 years ago | |

Thanks - that's an interesting article, although the key quote from the original paper doesn't really help make the case:

> The only thing that can be said with any certainty, is that not doing any technical debt maintenance does impact them negatively. But there isn’t a lot of difference from that point on. This is slightly controversial, and could be unrelated altogether.

Judging by the lack of activity on this thread, and the above article, I think I could effectively argue that technical debt either doesn't exist (outside the minds of developers), or that it's 'a thing', but just not that important.

This isn't what I was hoping to see...