But what about the bus factor?(michaelbromley.co.uk) |
But what about the bus factor?(michaelbromley.co.uk) |
I miraculously survived the accident with many broken bones and got back to work in about 3 months. And now it puts a smile on my face whenever I see the term "bus factor". And it probably scared our company quite a bit because I was the only software engineer, so definitely a bus factor of 1. But our company is mainly not a software company so they managed to survive with minimal support from me until I got back.
So yeah, it can happen! Be careful when crossing the street!
If you see enough companies over a long enough time these risks materialize with great regularity. Common reasons: burn out (probably #1), underpaid and undervalued so easily 'poached' (I don't like that term but don't have a better one),chronic illness, accidents, death. The latter is fortunately rare, two instances across 200+ companies across 15 years.
But the other ones happen frequently enough to take them serious. Most companies, unless they are very small are usually able to mitigate this to a very large degree without breaking the bank.
How about just saying that they left? 'Poached' puts the agency in the hands of their new employer, as if the person was cattle and their job change was due to cattle rustlin'. (Not that some management doesn't look at it that way.)
It was really a tragic occurrence, and very sad - Curtis was a great engineer. I think my drink tonight will be to him.
After a team member left and a new maintainer had to step in to maintain an unfamiliar code base, our new EM decided that the bus factor must be raised across all projects. So every quarter we shifted responsibilities to new projects and had someone else take over ownership - the idea being that the previous person would still support and we'd not be in trouble in case someone left. That's not what happened though.
It didn't really matter if the old owner was around to support. Productivity plummeted with every project. It was as if every owner of every project had left the team every quarter.
Now I'm more convinced than ever that it's better to pay the price of an owner leaving if and when it happens, rather than try to prevent or mitigate that risk up front. The productivity costs are far too high for most software products.
I would expect this to get responses like "you need to take your phone with you on your holidays" or "you should plan your holidays so that they don't happen at the time when the application is most needed".
In my experience, having the bus factor of one was not a problem for the company, it was a problem for me, because I needed to do a lot of negotiation about when to take my holidays. (After a few years I gave them the notice, and then it suddenly became a problem for the company.)
The lower the bus factor, the better the project. Raising the bus-factor to fight the bus-factor will lead to worse projects.
You can easily see that with popular open source projects. The more committers and managers you get, the more fighting, re-architecturing for the worse and harmful meta discussions how a project should be led. Eg COC. Good projects are usually led by 1, max 2. Everything above that is critical. If the one is hit by bus, another one can take over eventually. But not before. And it might need 10 years to find the one.
However, many senior developers recoil at the thought of quality documentation and prefer “self documenting” code which doesn’t cover the why and thought process behind the code. This makes something other than a full rewrite daunting. Even with the best self documenting code you will continue to be surprised in large projects. The full rewrite allows the rewriter to reexperience the process but it’s very expensive and may be worse than the starting point.
If science and engineering were done like software, so much knowledge would be lost. We know that there are giants in the space of Science, Einstein’s work was published and continued on by others to this day. I think the difference is that in science the motivation is to learn and teach, whereas in software people are pressed to create almost exclusively. We won’t publish papers, but to get closer to science, I think it’s necessary for solo developers to keep a development journal as has been done by scientists for some time.
I mention nuts and bolts engineering, because in engineering, the best are usually mentoring others, catching mistakes, and teaching rather than actually doing the legwork. That’s an alternative angle to fight the bus factor.
The number of people who, if hit by a bus and incapacitated, would allow an initiative to move forward.
…
On second thought, the average bus factor might be less than one
That’s OK, because most were written for my own use, in other projects.
The one project that I wrote, that is used by thousands, around the world, has been taken over by a new team, and I hardly have anything to do with it, anymore. It took ten years, of running the project alone, to get to that place.
In all my projects, these days, I develop them as if they are major corporate initiatives. I believe this makes it easier for others to use and understand (I also tend to use the MIT License).
Far too many organizations don't pay attention to operational risks...
[1] https://dsebastien.net/blog/2020-05-13-whats-the-bus-factor-...
In my experience this works great. Context switching is a problem, but there are ways to minimize the impact (similar tech stacks, good work organization, clearly defined priorities,...).
> Instead of 2 devs working for 2 months each on their own project, put them together on the first project for a month, then on the other project for the second month (simplified example, of course).
We did try this and it sounds good in theory but as the old saying goes 'two women can't have a baby in 4.5 months'.
In my experience (in this particular team and org) it would've simply been better to invest some time into documentation and suffer a short drop in productivity on a project when owners left.
One of the biggest barriers to that (besides teams needing guidance on how to do it) is conflict of interest, which is a factor in the why. I started saying it upfront with founders at my last company, when speaking of who we wanted to hire, and how engineering should be done: "One of the jobs of an engineer is to 'document themselves out of a job'." Then I add something like "And the company has to realize that engineers who can and will do that are generally the best engineers, and extremely valuable to the company."
Though, if the engineers didn't trust management (e.g., not exhibiting analogous professionalism and duty, such only looking at near-term selfish impact), then I can't argue that engineers should go out of their way to be easier for a stereotypical anti-leader to treat poorly. I'd have to appeal to the engineer's sense of professionalism, or of some degree of duty to other team members. I'd also know it's probably only a stopgap, and to give them whatever support I can if/when they look elsewhere for a home for an unusually great engineer.
That sounds nice in theory, but I've only ever seen it be a disaster in practice. It's like that mirror thing from Harry Potter. As soon as you actually try and make a maintainable code base, it starts going the other way.
The only way to truly optimise for onboarding is by writing the simplest, cleanest code possible. Everyone is already trying to do that by hiring the best devs they can. Any further efforts just look like more dependencies, more patterns, and more incidental complexity to act as guard rails in the pursuit of a newb-friendly codebase.
The end result is that new devs are productive at small and simple tasks, but big things take longer. Also, you're never going to find a manager that understands this trade off, so your new dev is going to be left in a terrible position trying to explain how they got so much work done in the first month when they knocked off all the small cards, but now the new feature that the business desperately needs has taken 3 months all by itself.
And it's also worth to pay the price to get the owner to not leave. So many companies seem to think penny-pinching is worth it, or only start giving out raises when actual attrition starts going up and many more people are already invisibly in the process of quitting.
This only generally happens if you have an environment where every project is a snowflake and the ramp-up time per project is thus high. If you standardize on a stack, then the internal organization of your codebases starts to look pretty similar, and there's less for people to ramp up on when they switch projects. Kill resume-driven-development at your company.
Because all I have are anecdotes, but all of my anecdotes are on the side of shared knowledge being heavily biased towards safety and consensus building demanding a lot of time. That happens even between two people that mostly agree, and only increase with more people or different opinions.
Of course, some times a bias towards safety is exactly what you want. But on informatics projects, that happens very rarely.
Once in a long while two people can have complementary ideas that make them faster. But it tends to happen more often when one is the clear owner of the project and the other is advising. Shared ownership seems to almost kill those chances.
why?
These days, I work harder than ever, for far better quality product, than I ever did, when a corporate drone. I don’t make a dime; but it’s never been about the money, for me.
I really love designing and shipping product. There’s a special joy that comes from architecting and finally delivering a complete software project. I consider it a craft, and take great pride in my craftsmanship.
Personally, I think it’s absolutely insane for corporations to deliberately ignore and insult folks like me. I was quite upset, when I first ran into it, but, upon reflection, I have to admit that it was probably the best thing that ever happened to me.
If my hand had not been forced, I would have kept on working, making other people money, and never having the joy that I experience, every day.
Definitely worth it. Would buy again.
Implementing such systems is absolutely a challenge as there is negative feedback to such things all the time; designing such a system, putting persons in place whose job it is to enforce the process, putting persons in place to weigh in on if an action or decision matches the letter and spirit of the process, all of this take a lot of discipline and maturity from the participating members; I've done this with several orgs now, and even with backing from the VP level, at best I get belligerent participants, and truly I cannot tell if their resistance is that they don't like the decisions we make, or if they just don't understand the process and want the ability to complain to someone directly.
Most of the issues you get with multiple stakeholders is because of a lack of discipline within the org; people who are too used to "just doing things" without really considering the full consequences of actions in favor of just doing what they want. You see this in basically every collective project, not just programming projects too. I consider it a win if we have people begrudgingly participating, but ultimately participating.
The belligerence is that there’s very little benefit career-wise to being a good committee member on low-visibility committees. At best you get some nominal recognition, and at worst it’s a distraction from the things that actually do further your career. Nobody gets a gold star for that kind of work, but you do get big promotions when you can own chunks of work yourself (even when done poorly).
And if a project stalls, he/she probably considers it done, in opposition to others. You can easily extend such a project under your own maintainance then, but then you must be capable of maintaining it.
ex: D3 was/is really mike bostock. the exception there has been, afaict, the GIS packages and maybe changed over last 1-2 years
ex: we use 'got' for JS HTTP, and most dev is 1-2 devs moving quickly, and community for QA
ex: caddy felt the same, but that may have also changed post acquisition
interestingly, all 3 support plugins/galleries for occasional non core contribs
the article was clever to observe that most commercial sw does bump up bus factor in a funny way: sustainable bump.. but typically goes up by only a little!
Could of hard-won experiences:
- under no ciscumstances should you do the rewrite until you understand the whole domain, which is not until you know the codebase inside-out. And even then it's probably a very bad idea ("Yes but the old version allowed us to do X...").
- start with business requirements: make sure you understand what people expect from the system, how they use it, what it does at high level. Sit with users and watch them use the app!
- tests: do not touch the code until you understand the existing tests or, if tests do not exist, until you have covered the code with your own, preferably integration or end-to-end tests (unit tests are great, but they don't guard you against integration failures, which is what you want to be protected against here too)
- skim the documentation if it exists, but don't put too much faith in it as it is probably out of date. Same for comments.
- slowly start fixing bugs, cleaning up some code... whatever you do, make sure it is covered with tests.
- good luck. :)
Right now I'm looking at a relatively small code base that was written by an engineer in the embedded systems team. The original authors are retired. It's generously documented, and a pleasure to read. This makes my life easier, but also less of a hassle for the un-retired engineers on the receiving end of my requests for help. I don't know about commercial coding, but adding comments and documentation can help a non-specialist work their way through code if they have some decent programming background.
The full re-write can happen if the departed developer is an entire departed business, e.g., an acquired company comes with code that is obfuscated to the point of being Goedel-undecidable ;-)
Science isn't flawless. There's the so called reproducibility crisis. I think what helps science is that nothing of lasting value depends on a single critical path from data to results, but is reinforced by looking at things from multiple angles. For instance we'd still trust general relativity had the Eddington experiment never happened. And scientists such as myself are always looking for ways to make our own work more open and reproducible. In my work, Jupyter Notebook has been a game changer.
> However, many senior developers recoil at the thought of quality documentation
There is no such a thing as quality documentation. An individual here and there is able and willing to write one. Average developer is not. And I don't even blame them, it is in fact skill and job occupation on itself to be able to do so.
Developers are not even trained in how to write good documentation. There are no processes to check, test and review documentation.
But often there are some or all of those pieces missing, and so yeah, code review ends up being a bit shallow and performative.
What matters is that there is a clear owner who is the voice of final decision. In most cases there is no objectively better way to do something, so having a person who is a tie-breaker is beneficial. Of course, they must be involved in actual coding, doing code reviews of other people's code and similar. And other people must review the owner's and each others code.
But, having an owner is not the same as having a bus factor 1. There should be an owner, but other people should know the project and the codebase enough to step in at any given time, even as (temporary) owners if needed. And this is only possible if they actually work on the project, even if the approach that the team (/owner) decides upon is not what they would pick.
Oh, and we don't do ownership thing either.
An example of the opposite I thought of immediately is neovim. The vim dev didn't really want too much changes so the neovim fork took of, and eventually vim development sped up too.
'few devs vs despite' matters a bit less when picking: it's the results + future expectation, irrespective of how :)
'how' is interesting . having observed dedicated oss devs owning a proj for years vs contracted devs working through tasks for their latest gig, I have unsurprising theories. Trickier is when going sustainable via revenue (which we do), how to keep that sense of ownership, vs usual employee code: industry increasingly adopts OSS practices like GitHub flow, but that misses much of the OSS maintainer magic.
Most of my OSS projects are 1-file UI widgets (but with a lot more testing code than implementation code). I do, however, have a couple of monsters.
Because of my experience (which includes many mistakes), I am able to develop projects of a far more ambitious scope than the run-of-the-mill developer. The one that I'm working on now (not OSS), for example, is one that would usually be done by a heterogenous team.
See also: documentation, testing, refactoring and all the other goodies that keep a large codebase maintainable.
I agree that not all environments are good at managing developers though (and that is probably a huge understatement).
Simply said, that deadlines and OKRs don't account for bus factor.
If you are in a place with code-review, the 2-person system allows one person to at least read the code that is being written as it's being written, so the genesis and reasoning behind such code might be a little more obvious.
At 62, I can relate.
I think you might enjoy this short parable, as told by Alan Watts:
Quotes:
> Jiro Ono: Once you decide on your occupation... you must immerse yourself in your work. You have to fall in love with your work. Never complain about your job. You must dedicate your life to mastering your skill. That's the secret of success... and is the key to being regarded honorably.
and
> Jiro Ono: I do the same thing over and over, improving bit by bit. There is always a yearning to achieve more. I'll continue to climb, trying to reach the top, but no one knows where the top is.
[1] https://www.imdb.com/title/tt1772925/
[2] https://www.imdb.com/title/tt1772925/quotes/?ref_=tt_trv_qu
That is such a fantastic luxury position to be in.
I worked hard to get here. Saved 25-40% of my money, while working. Lived real "low on the hog."
It's "f**k you money." I just thought that the idea was for me to say "f**k you" to employers; not the other way 'round (which is what happened).
It's a bit like when they want to know if you've family/responsabilities, I doubt it is to pay you a salary going with those, but more to gauge if you'll be a good slave or not...
Anyways, if I was in your position, I would do whatever I want, but without working for a company anyways, is not fuck you money (or attitude) at leaat, if you still have to obey the boss on all the bullshit right?
Best of luck!
PS: Oh now I saw your username, I've read you a lot arund here Chris, I think you're doing great with your own libraries/iOS stuff, I don't see why you might need to apply to a company, maybe you should start yours instead. You would make a good leader
Does he manage every aspect of his restaurant chain? Does he install all the plumbing and electrics? Does he wash all the dishes?
In 'software', it seems the 'skill' you're supposed to 'master' is always learning more of everything all the time.
I can not be a master of everything. I can not keep up with docker/k8s/security/design/css/UI/accessibility/i18n while continuing to 'achieve more' with SQL, server-side development, documentation, testing and more.
There are physical limits (x hours in a day, ability to focus and concentrate, and new 'things' constantly coming out).
I say this because as 'good' as I may be in the 'latest' in, say, 4 areas of development... I'm going to be judged harshly by certain groups of folks (team/hiring folks) because, for example, I may still be using bootstrap 5, or angular instead of react, or whatnot, regardless of if that's my focus or not.
I've fallen in love with some of my work, but there's a limit to how much I can do, and a limit to how much I can actually love. Some I just tolerate.
I was a manager for many years (you can see many testimonials on my LinkedIN page). I just hated the job, although the consensus seemed to be that I was pretty good at it.
I do have a couple of companies, but they are more so I can access resources that are not available to individuals. I am currently working with a small 501(c)(3) (US nonprofit). It is a team, but there's only a couple of us tecchies, and I do most of the coding.