The problem that the article doesn't address is that users don't actually seem to mind using terrible software so long as it solves the problem they face better than not using it. I could list literally hundreds of half-assed, broken, bloated applications that I've encountered in the past 25 years that have done very well simply because they kind of solve a problem a bit for the user.

Pushing out something completely broken that doesn't do what it's supposed to is definitely not going to work (duh!). Pushing out an app that solves the problem of managing shopping lists that has a bug where it doesn't work given a particular set of circumstances will still lead to many people using it if the users don't have any alternatives and it's better than using a piece of paper.

Software quality is important to companies because it means that they can spend more time building features instead of fighting fires, and because low quality represents a threat that a competitor could launch a better, less buggy app. Users mostly don't care so long as the app works well enough to do what they need it to do (but they're not dumb, they'll still pick the least buggy option if there are alternatives..).

A high level of quality in software is not important unless you're entering an already well-served market. I wish it was.

It seems strange to me that software engineers are so frequently singled out for schedule slippage, when the impression I get is that every novel engineering project suffers from the exact same problem. Projects to design and build new military and civilian hardware and infrastructure always involve budget and schedule overruns of months or years. Can anyone provide convincing empirical evidence that software projects are delayed more than hardware projects of equivalent budget and distinctiveness from previous solutions?

A negative comparison often seems to be made between software engineering and construction, but it seems to me that the latter is a unique subfield of engineering, where you have an unusually large number of projects with a roughly homogenous set of constraints and variables. This has allowed those constraints and variables to be studied, understood and mastered to produce a discipline that more closely resembles mass-production. And in those subfields of software that also involve more homogenous constraints, such as the production of standard commercial websites, you do see a more controlled and templated approach, using tools like Wordpress.

When I was in the VFX world we had fixed, hard deadlines. Once the release date is printed on a movie poster it's not changing. Some parts of the process were easy to estimate since they'd been done many times before, but on every film there were some experimental new things. Often they would just end up with 2 or 3 different teams working on the same problem. One with a known technique, and one or two trying something new. Then if the new stuff didn't work or was taking too long, you could always fall back to the old way. Which wouldn't look as good or achieve the effect you wanted, but :shrug:. There was always the chance that you'd work sometimes for years though and your work would get dropped which was never that fun for the artists involved. Most development teams don't have the resources to have an A and B team working on the same problem but it's a way to contain risk.

The studios use release dates as deadlines basically to contain costs. Otherwise some directors will just keep working on a movie forever (or keep going back to it like George Lucas). Some movies, especially with VFX, are never really done, they just get to the deadline and finish with what they have at that point.

Alternate view point - driving projects to a specific deadline allows the other departments in the company to coordinate marketing, packaging, and selling the product. These people don't sit on their hands patiently waiting for engineers to finish building, their work can take many months just as the development does, and sometimes also involves making tradeoffs to deliver on time. No company wants to wait another year after development is finished before they start earning money for it.

There is nothing wrong with working to dead lines estimating and planing development and releasing software thats good enough not perfect.

The problem is when people start trying to change the plan half way through execution, non tech people will constantly try to do this because they are clueless its the project leads job to tell them no.

When your manager asks you "if anything can be done to pull those dates in." you say "no" problem solved, until your line manager goes to the big boss who asks the same thing and do you know what your answer is? Thats right its "no". When the big boss says you have to do something then tell him "I cant magic more time you need to cut features", big boss and manager will argue their point but unless you can do magic your point remains the same.

I know its scary being blunt with your big boss but as a senior developer thats your job.

There is a counter intuitive thing about software project estimation that took a long time for me to discover.

The more rigorously you try to analyse the problem, cutting it into smaller and smaller parts, the more error you introduce, reducing the value of the estimate at each step.

Thus, the best practice is to give a very rough estimate based on the scale of the project and your past experiences.

If you don't have previous experience with similar projects, then you should not even try to estimate.

Once you've been around in this software business for long enough you see the same articles written over and over again... there will always be articles about estimating software, yet there is no solution to this problem.

Anyhow, it doesn't matter. Software devs are the lowest cog, they don't get to extend the delivery date. All the layers above - management, marketing, sales etc need a date to work to to deliver their work, and they generate the $$$. No-one cares if the software is low quality, as long as it is delivered. Dev team can do further updates to bug fix after delivery. The end.

I'm not convinced an arbitrary date is a bad idea, and in fact, I continue to think it's a good one. If you don't have a point in time you are optimizing for as a developer, then why wouldn't you continue to gold plate and improve your code? Once you have a ship date, you know when it is time to knock it off and start heading downhill.

The real problem here is the leadership deficit. The manager got pushback on the estimates, and instead of explaining the reasoning and bargaining on scope, he caved and kicked the can down the road by letting the estimates crumble. Yes, estimating is hard, and you are probably going to be wrong, but once you have your finger on the scale to get a desired result, you can't blame that on the difficulty of estimating. Just deplorable leadership.

The article is full with interesting nuggets of wisdom, but I found it all over the place, as it mixes several issues (all of them important) into one.

The article mentiones "high-value" projects (vs "low-value" projects), but the actual distinction made is between high-effort (new framework, new kind of product) and low-effort. Value and effort are not neccessarily correlated, indeed by the end of the article the team has had a high-effort project that resulted in a low-value product.

It also conflates product discovery (what will actually have value to the users) and technical discovery (how do we build the damn thing). It starts with the technical challenges (and I totally subscribe to the notion that estimating anything that you're not doing routinely is Dangerous, see https://blog.senko.net/bridges-vs-apps) but then veers off into describing mismatch between features built and value add (also a very important subject, but a different one).

I would love to see author expand each of the sections into an own article and explore these things in more depth!

The most underwhelming part for me are the recommendations at the conclusion. So (focusing here on tech aspects) if "Something by Date (tm)" methodology is bad (which I can agree with in principle), the solution to "let competent engineers prototype [the app] in 3 or 4 different frameworks", ie . "give your engineers unbounded time"? All developers (and I am one) wish they'd have more time to implement things properly, so that's not a new insight. But it's not actionable either - "give more time" then turns into "Something by Later Date (tm)", which is basically the same problem, just with more money (and thus stakes) involved.

I hoped to see an exploration into how to creatively solve for "can we deliver value" + "we have this budget and time" constraints.

Chapter 11. Plan to Throw One Away

Chemical engineers learned long ago that a process that works in the laboratory cannot be implemented in a factory in only one step. An intermediate step called the pilot plant is necessary to give experience in scaling quantities up and in operating in nonprotective environments. For example, a laboratory process for desalting water will be tested in a pilot plant of 10,000 gallon/day capacity before being used for a 2,000,000 gallon/day community water system.

Programming system builders have also been exposed to this lesson, but it seems to have not yet been learned. Project after project designs a set of algorithms and then plunges into construction of customer-deliverable software on a schedule that demands delivery of the first thing built.

In most projects, the first system built is barely usable. It may be too slow, too big, awkward to use, or all three. There is no alternative but to start again, smarting but smarter, and build a redesigned version in which these problems are solved. The discard and redesign may be done in one lump, or it may be done piece-by-piece. But all large-system experience shows that it will be done.[2] Where a new system concept or new technology is used, one has to build a system to throw away, for even the best planning is not so omniscient as to get it right the first time.

The management question, therefore, is not whether to build a pilot system and throw it away. You will do that. The only question is whether to plan in advance to build a throwaway, or to promise to deliver the throwaway to customers. Seen this way, the answer is much clearer. Delivering that throwaway to customers buys time, but it does so only at the cost of agony for the user, distraction for the builders while they do the redesign, and a bad reputation for the product that the best redesign will find hard to live down.

Hence plan to throw one away; you will, anyhow.

-- Frederick Brooks, The Mythical Man-Month

Which is more likely: Businesses invent truly arbitrary deadlines for the hell of it, or the “engineers” don’t want to pay attention to business requirements and competitive pressures?

When so-called engineers stop spending half the development schedule choosing a framework and the other half trying to make their dev setup work on everyone’s personalized laptop they will have some credibility complaining about “arbitrary” business goals and requirements.

Well the counter argument is that shipping later also destroys value (because value can only actually start to be accumulated once a product has shipped).

But this article doesn't seem to address the obvious counter-point.

For instance, take the below:

> wouldn't you rather run a your customer's payments through code that is at least attempting to handle error conditions, rather than some happy path code that just assumes everything works?

But this doesn't address the obvious issue which is, there is value destroyed (or not created) by not allowing customers to pay you earlier.

"Our boss can't judge the quality of our work, but he knows when it's late" -- Dilbert

The whole point of organizing things in the first place, is to reduce or remove uncertainty and risk, which cannot be done with software projects.

Software engineering is similar to writing a book, or a movie script; but it is mistaken for some advanced reorganization of numbers in an Excel sheet.

Normal leadership training relies heavily on deadlines to "move things forward", or force us to "see where we are" etc. This only creates stress on the creators themselves, making them far worse at actually creating something. This is something all developers know, and this is why companies started by developers often have a very relaxed attitude towards work. I.e. the dart board, pool table, or beer taps are not there to make the office "look cool", it's there to make a relaxed atmosphere, where you work when you are "in the zone", and relax when you are not.

Everyone who writes movie scripts, books or software knows that you can do more in 4 hours of when you are in "the zone", than in 80 hours in an open landscape, where days are split into short work hours, disturbed by status meetings about progress etc.

Work expands to fill the time you have. If you don't set a goal, you'll never ship. Estimation is not an exact science, but, I rather set a goal post that I can move if I need to than not have one at all.

There are three kinds of deadlines: fake deadlines, contract/revenue driven deadlines and reality driven deadlines. Most problems with deadlines I've seen come from chronically mislabeling the deadline (e.g. sales don't close, so engineers start ignoring revenue driven deadlines). Fake deadlines sometimes are necessary to get things to some finished state, but they should be used very sparingly. Deadlines should be honest, and when something changes (e.g. some other team is going to be 3 weeks late on their part of the product so being on time doesn't matter) the deadlines should be quickly adjusted. Deadlines are a work-planning and asset allocation constraint, and will cause very bad decisions if they are not understood and managed well.

A better solution ten years later might not be a solution at all when running a business. So it cannot be just open ended.

The point I take away from this more: poor planning and execution processes destroy value. It is not an "us" vs "them". Have clear responsibilities for removing internal/political blockages, for allowing testing, etc. How to manage new delays and unexpected issues? Who can decide on changes? Organizations can manage to hit target with a reasonable product/outcome.

this seems to mainly be relating to pushing to an arbitrary date for creating something new but still sometimes dates are given to us because of some sort of regulatory reason or a company contract meaning that something needs to be done by a certain time - before anyone says don't make contracts like that: The Danish and Swedish parts of Thomson Reuters WestLaw were sold off to an English holding company, as part of the sale the sold off parts were allowed to remain on WestLaw for one year after which they would have had to pay approx. $165,000 dollars a month.

Obviously we got off of by that arbitrary date, and if we hadn't that would have been a value destroying mistake.

on edit: the weird monetary amount is my attempt to turn to U.S.D the DKK amount at the time which was 1 million dkk - iirc the dollar was at a low but it was between 161 - 165.

To be fair, Engineers engineer and wouldn't mind spending twenty years engineering and discovering new paradigms and interesting things to get 1.2% gains. Deadlines force the issue to put the wandering mind of the engineer in a straight-ish path. Not perfect, but the alternative would take a lifetime.

There is never going to be one clear way to deal with scheduling around software projects. The number of variables involved is vast.

I view schedules around software projects as a few different things:

- Public expectations for specific deliverables that your customers can align to, many times with a calendar unique to each customer.

- Internal carrot/stick for managing constraints around active feature development. If you have an infinity budget (film/AAA gaming), you might be able to run less-bounded parallel efforts as noted elsewhere in this thread.

- Orthogonal operational concerns (planned outages, et. al.)

- Roadmap for strategic product development that the investors can think about

So, when someone says something to me like "are we still on track for end of the week?", I have to provide an extremely qualified series of responses and ask more questions.

I always tell my PM:

You can pick a feature set or a launch date. You can't pick both.

I’ve lived through a project managed like this. The problem isn’t the date. It’s the unwillingness to prioritize and cut scope when the bottoms up estimates show your first version is too big.

> The Big Boss is a pressure kind of guy, he believes that people work best when they are under pressure and if pressure isn't applied, they waste a lot of time on unnecessary things. [...] he goes away thinking what he already knew, that engineering is just screwing around instead of getting the project done.

Sadly, he's often not wrong. My own work ethic isn't exemplary, and sometimes in the absence of pressure I slack off. I'm surely not the only one. I don't know what to do about it though.

To list just a few such slipping projects outside the software realm:

* Space Launch System

* Boston's Big Dig

* New Berlin-Brandenburg Airport

I think there's a difference between companies that build software for the market versus those that build software for a contract. The former tends to be driven by a team that believes in the product that they're trying to deliver, has a management team that has at least some vision for the software and investors that believe in the company's approach to the market. They want to make software / a product that represents the organization well and has long term value.

In the latter case, like so many government sponsored engineering projects, the company has no real long term incentive beyond what is written into the contract. Management isn't driven by the software being produced so much as meeting the contractual requirements that result in payments. In my experience, software contracting houses are a lot more likely to hold schedules over the heads of developers, and that's purely driven by the terms of the contract (billable hours). In contrast, even when I have been a contractor for a company that is building the product for itself, they tend to push for the goals of the project while ensuring it meets the needs of the company. The difference in pressure as a developer is quite substantial.

> It seems strange to me that software engineers are so frequently singled out for schedule slippage

It follows this trend I've seen across the board in business throughout my career, of mistaking the last hair on the last yak for the entire job. The classic example is documentation. "Hey the electrical drawings are due on Thursday the Aprilteenth, can you have them ready by then?" Oh sure, that's two months away, that's plenty of time to bang out some drawings, right? But wait! The drawings depend on the functional specification. And the functional specification depends on the functional description. And the functional description is blocked on some TQs from the client. And you're still expected to produce these drawings by Thursday Aprilteenth when realistically you're not going to see even a first draft of the functional specification until the Wtf'th of August. But now it's your fault that the drawings are late because you weren't paying attention and you agreed to a due date that didn't depend on your dependencies.

I take this one particularly personally because I'm in industrial automation and, almost by definition, we get the last bite of the pie on every single project. Sure, there are some things we can start early but on the project-specific parts, generally we're waiting for everyone else to do their job before we get priority.

"Can you help us with the motion control software for AwesomeBot?"

Sure, no worries!

"Just remember we need AwesomeBot running by Thursday."

Hey cool, you present me with an AwesomeBot-shaped shell and I'll fill it with closed-loop-ey goodness.

"It's Friday, why isn't AwesomeBot doing somersaults like you told us it would?"

Uh well, AwesomeBot is lying disassembled on the mechanical workbench and half the laser cut sheet metal hasn't arrived.

"But you told us Thursday! Okay, we'll finish putting AwesomeBot together and let you know when it's done."

<fast forward 17 months>

"Hi remember us? We're AwesomeBot inc."

Oh hey how's it going with that?

"We've got AwesomeBot fully assembled and we're ready to go! Can you come down tomorrow and install the software on it so it does everything we ever dreamed of including anything we might have imagined it might do in the 17 months since we last saw you?"

Uhhhh... so how's testing going, have you switched it on yet?

"No, why would we do that? We were only waiting for you!"

Yeah, this is an unfortunate truth that Agile has to confront; there may be hard coordination deadlines. Or, in startups, a financial "runway".

On the other hand, setting a deadline can't force something to be possible, it can only force people to work harder and more painfully towards it. I'm sure the Amazon drone delivery failure had a date target, for example. And there have been plenty of failed "big bang" IT migrations delivered by similar immovable deadlines.

You have to try to understand that building software is not like baking a cake in a custom shape. Which if you have seen those shows it requires a certain amount of ingenuity to create a cake shaped like a fountain. Or whatever.

But fundamentally software development has 1000 to 1000000 more moving pieces than a cake, generally a research project, like building a new type of aircraft engine. Except, imagine it was a really new type of engine that was being built based on an alien landing. But they could not get into the engine to see how it worked because it was an unknown alloy. So they were trying to invent a new type of anti-gravity drive based on a few clues.

I mean we are not trying to defy gravity, but there a shitton of unknown unsolved problems. You can't promise or create marketing for a feature unless we already know it's possible. For starters.

But the issue is worse than that. What generally happens is once one part of the new engine is barely working, they demand another new invention that has to be completes in the same deadline before the other one has corner cases examined even.

Furthermore, financial penalties on late delivery are not uncommon in enterprise business-to-business settings. These penalties are sometimes quite steep

If the choice is between releasing kinda-working, but unpolished software, or throwing a developer's monthly labor cost out of the window every single day, these management anti-patterns suddenly make much more sense

The problem is that nothing is ever so clear cut and dry and it's always _possible_ to cut corners to rush something out, but rarely is it the right thing to do. Everyone should brush their teeth and look both ways before crossing the street, but when being chased by a killer it might make sense to skip these things. However that should be extremely rare.

The challenge is that there are no hard rules and it's highly dependent on the context. That requires mutual understanding and trust between the business and development.

> I know its scary being blunt with your big boss but as a senior developer thats your job.

This. They could hire any random chump to say they're right. If you're the expert, they hired you to tell them whether they're right.

Of course, not all managers understand this but the sooner you realise you're working for one who doesn't, the better.

Ah, you are one of those "difficult developers" :) Be blunt with your boss but also dont forget to quit if they dont change their behaviors.

Just "finishing up" a project of "6 months" where we went from reasonable to unreasonable scope, and I was the only one that said it was unlikely we were going to finish on time, weird that I was the only non-contractor....

Almost a year a shoddy version that's just copying data from their existing product is only mostly a buggy mess!

The issue is that many people are in a situation where the manager will either decide to just not listen and promise those dates or features anyway or they will be demoted,.fired or otherwise penalized for not going along with it.

That's a problem because many people do not have the luxury of quitting for another job easily.

Well, sometimes you can reduce scope to meet a deadline. It's often not advisable, but it's also sometimes necessary.

The magical scope creep fairy wants to talk to you. :)

It’s funny you say this because every software project estimate I’ve ever done has basically been to invent lots of small bricks and fiddle the numbers of these so the total matches what seems to be the intuitive size

Theoretically, this should be the exact opposite of the truth. Breaking a big prediction into a whole lot of smaller predictions is the basic intuition behind Fermi estimation and wisdom of crowds. As long as errors are symmetrically distributed and independent of each error, they'll cancel each other in aggregate.

The problem is estimation errors are not symmetrically distributed because engineers chronically underestimate how long something will take, and the problem is exacerbated by management pressure giving them an incentive to estimate even lower.

Well it sounds like you are not doing Agile correctly.

Jokes aside I think accumulation of errors together the fact that time estimations are hard and Parkinson's law is the fatal flaws with Agile.

> Thus, the best practice is to give a very rough estimate based on the scale of the project and your past experiences.

This is my experience too. It is somehow easier to estimate big scopes then many small that sums up to the same scope. Also you get a better feeling of how far you have come when looking at the big scope then when digging down to the details.

My formula is usually to listen to the project idea, talk to a few people, take my first feeling how long it could take and then multiply by 5. This usually is pretty accurate.

Do you want developers that are passionate about their work or do you want compliant drones that will only ever do the bare minimum for you and blindly follow the directions laid out by management, even when they know it will be a disaster? I see a lot of companies where management just keep pushing their employees to get a month's work done in a few days, cutting corners and creating issues because nothing is properly thought out or tested.

It's called "development" not "Bam, it's done!" The best companies allow proper time for the development process to take place and have a proper team structure. Companies where everything is discussed as a team and you succeed and fail as a team. Proper agile teams which communicate at every step, with developers, testers, a scrum master, a business owner and a manager.

If you don't have that, you're basically not a proper development team, you're more of a startup that can't afford a proper development team. Nothing wrong with startups, but if you are a startup then you have to deal with your own weaknesses and shortcomings and take your failures on the chin.

From anecdotal evidence, a) is more likely than b). The deadlines are arbitrary because most companies are slow bureaucratic oil tankers that are difficult to steer and will mow down everything in their path once a course is set. Management's job should to be to work on this, to improve the system as a whole and making it more nimble, not on myopic deadlines that fulfil some checkbox and are tied to your EOY bonus. This is outlined by people like Deming and others in the systems thinking space.

From experience, in their career, engineers not only need to excel technically, but are also forced to pick up everything from UX, methodological BS (from Scrum to Itil) to domain specific know-how in multiple fields or areas. Since many managers do not know what they're doing, senior engineers often times end up being de facto management consultants as well. If you are working in a business environment (as opposed to writing low-level drivers or whatever) it's almost impossible to not pick up on what's going on around you.

How many banking managers know how to code? How many engineers working in banking know at least something about the processes, compliance issues, how the org is structured, what the competitors are?

The stereotype of the autistic programmer who is only interested in shiny gadgets and tech needs to die.

I often see engineers having a cynical self defeating view of the business. They will blindly follow complex requirements while complaining that there is not much they can do about it.

But if you demonstrate that you understand things from the business perspective, they are much more receptive to making adjustments.

Product managers like to gold plate things as much as engineers. There is usually some fat you can trim from the scope to make a deadline.

Businesses—in the guise of mediocre managers—absolutely invent arbitrary deadlines as a mechanism of control. If the deadline were real, it would have context, engineers could use that context to adjust the scope and design of what they're building and we wouldn't be having these conversations in the first place.

The problem begins when deadlines are used to limit agency rather than support it.

I hear you, but this problem becomes exponentially harder as your engineering org grows.

I witness on a daily basis PRs that have no body getting merged with absolutely zero comments and a blanket approval as long as it passes our (broken) CI pipeline. I witness obvious poor quality in the code, but engineers want to seem like they are working and will just blanket approve PRs, while i'm in the middle of writing up my code review denying the PR.

If you are a developer on a team and want your codebase to be high quality, you end up no longer writing the code and instead spend all of your time gatekeeping via code reviews. This leads to burn out.

I wonder if the nuance being identified by the other commenters might be the tragedy of the commons that can play out in collaborative settings. I'm right there with you, btw. Mine is also a non-sexy, non-commercial app and I'm happily a one man band. But a younger me, a junior member of a team, under pressure without close supervision would probably commit a certain amount of half baked spaghetti and be part of the governance problem.