Boeing's 737 MAX Software Outsourced to $9-an-Hour Engineers (2019)(industryweek.com) |
Boeing's 737 MAX Software Outsourced to $9-an-Hour Engineers (2019)(industryweek.com) |
If you parcel out parts of your job in something as critical as Airplane Construction, it is your responsibility to validate all specifications/testing/verification/etc. If you treat "Airplane Engineering" as mere "Embedded Engineering" you need to be put out of business.
The article explicitly says MCAS and "cockpit warning light not working" problems (which were the cause of the two crashes) were NOT outsourced.
Key points;
“Boeing was doing all kinds of things, everything you can imagine, to reduce cost, including moving work from Puget Sound, because we’d become very expensive here,” said Rick Ludtke, a former Boeing flight controls engineer laid off in 2017. “All that’s very understandable if you think of it from a business perspective. Slowly over time it appears that’s eroded the ability for Puget Sound designers to design.
Rabin, the former software engineer, recalled one manager saying at an all-hands meeting that Boeing didn’t need senior engineers because its products were mature. “I was shocked that in a room full of a couple hundred mostly senior engineers we were being told that we weren’t needed,” said Rabin, who was laid off in 2015.
“Engineering started becoming a commodity,” said Vance Hilderman, who co-founded a company called TekSci that supplied aerospace contract engineers and began losing work to overseas competitors in the early 2000s.
The last point is the most damning; Airplane Engineering cannot be commodity engineering.
While i understand your point, the headline gives the impression that Boeing's problems stem from outsourcing which then emboldens the racists to come out of the woodwork (as can be seen from some of the downvoted comments here) and start manipulating the conversations in a different and unproductive direction. Some of the folks commenting didn't even read the article before dunking on the outsourced country/engineers. It is highly frustrating when this sort of thing happens and hence my warning in the first line of my response to ignore the title and read the article.
It would be great if you folks (maybe in collaboration with others sources/folks) could collate all the available info. on the disaster that is now "Boeing Engineering" (HW/SW) and publish a point-by-point synopsis of everything known until now. This would be a great learning lesson to the entire Industry on what not to do in pursuit of mere profits.
Shame. Industrial companies, if you decide as pincher, you will be doomed in your little world... as it always has been :)
It works for them, because like Boeing said, they have old battles tested processes which usually means big mistakes can easily get caught ahead of time. And if big mistakes don't get caught they usually can't kill anyone and generate a huge public scandal.
What I think actually happens is they get to prove all sorts of ridiculous things are good management and that all sorts of managers need to be coming up with complex ways to manage what would be obvious in a situation where reality was still visible underneath.
In the end they pay a lot more than if they had competent individual contributors and less darwinian fights to the death in their org charts. But having a better organization is not really compatible with the incentive system involved.
As I understand it, the software performed exactly as designed, forcing the nose of the plane downwards to counter the upwards torque produced by the engines being offset relative to the centre of gravity of the plane.
Mentour Pilot had a great episode on the two tragic accidents. It appears that Boeing did not expose the way the software was designed to behave, and the system silently turned itself back on even after the pilots disengaged it.
-Gilfoyle
Seriously? I mean, could you still order anything but a Max from Boeing in that size? And that being the most useful size, of course it became the top seller. Because it was the only offering.
Whereas, the best performing companies hire engineers who are empowered to understand the goals, understand the problem, and push back when appropriate.
The results are much better, because the people working on each system, understand it deeply and care.
A low-cost contractor isn’t allowed to care. They aren’t even allowed to talk to somebody who matters.
This is a management disgrace.
Many managers far prefer the latter over the former.
The theme throughout has been cost cutting and poor oversight for the sole benefit of shareholder profits.
A proper in house team would probably say, hey what happens when the sole sensor feeding our software is obstructed by a birthday balloon? Maybe we need an exponentially backoff instead of every 30 seconds. Maybe nose down actions shouldn't be automatically executed at low altitude.
What we see playing out is the result of delegation and "not my problem", outsourcing safety critical systems to the lowest bidder...
Large companies are mechanisms that run on their own. People have limited agency in a lot of situations but for big decisions ("our company's only real product is fundamentally dogshit, lets start over") there literally are no people on earth who can actually make them. They are pre-made by the structure of the company, capital markets, and historical accident.
However, the offshore team are usually not intended to act as domain experts. In fact, they were very likely explicitly proscribed from interpreting the specifications handed to them to guard against them trying to act as domain experts and delivering something different from expectations.
As such, they were (likely) not the ones who specified that MCAS should silently turn itself on after a pilot turned it off. That misjudgment probably lies with the engineering team who made that design decision, and it had tragic results.
I have been to IBM offices in São Paulo a few times and let me just say, although they paid reasonably okay salaries it wasn't the first choice of work for most engineers...
And that is on top of the problems you are describing.
This is very likely correct.
People on HN don't seem to understand how software is created in regulated industries outside the tech bubble.
In a complex system such an aircraft, the behavior of the system is modeled and detailed requirements are generated.
These models are created by the system engineers.
The models and requirements are then handed off to the software engineers to implement the modeled behaviour in code.
The software is then tested to see if the behaviour matches the models.
So it doesn't matter how much the SWEs were paid, if the software met the requirements, and implemented the models as designed, then the software engineers did their jobs.
What if sensor malfunctions?, how should software act?, will it crash the plane?
If yes, then raise problem and refuse to implement / avoid killing people.
Then you have QA which should have the same questions and test plan to make sure corner cases are covered. QA is also responsible.
Then you have management which is supposed to make sure there are people skilled enough doing the above and the initial systems parts and encourage this environment.
Then you have business people to ensure sufficient resources to hire competent people above.
So what is happening is a massive problem everywhere...
Throw in a massive timezone difference and every misunderstanding winds up living in your system.
The problem was that they didn't outsource the design spec. I doubt an externally developed design would be so contrary, with a flippant we'll document it and include it in the short brush up training.
An individual contributor is typically given a tangible task (eg: deliver a product that does x in y months for a cost of z). There are several qualified people they could choose from to achieve this task, which drives down the cost. I worked for a company with 10,000 engineers. Oversimplifying for effect, but their skills are not unique.
A leader is given a far more intangible task (eg: gain 30% market share in z years for your division OR deliver 9M of product per day through 202X), and they are taking a huge risk. The team under them may not have the slightest clue how to do this. The interest rates can change. Currency fluctuates. They are paid to calmly chart a path through the chaos and deliver a result. At this level, there were about 60 of us, and the atmosphere gets rarer as you rise further in the ranks. They made absolutely certain we were kept super happy because we could create tangible value orders of magnitude above the average employee, and they REALLY did not want us working for a competitor.
https://simpleflying.com/avianca-airbus-a319-balloon-strike/
https://www.latimes.com/archives/la-xpm-1994-09-22-me-41720-...
https://dronedj.com/2021/08/31/update-faa-says-envoy-air-pas...
But we really need to pump the brakes on the whole “everything must be in the passive voice and overly euphemized to the point of nigh-incomprehensibility”. If anything, it contributes to the click-bait problem.
I see: “Boeing outsourced to $9/hr SWE” and immediately start thinking about a dangerous corporate culture that prioritizes profits over lives. I’d argue most people probably interpret the headline in a similar fashion.
Just because some already-inclined reactionaries wrongly jump to “India lol” doesn’t mean the publication has some moral responsibility to contort the headline into something less effective that will get even more dramatically outcompeted by click bait headlines that actually bait racists (“Deadly Boeing crash was running Indian software”).
Outsourcing is a perfectly logical approach to saving costs. But the company doing it has to be careful on how it does it, have necessary modeling/tests/verification/QC/QA in place and be very very careful in outsourcing highly critical/specialized/domain specific subsystems (if at all doing it).
From all that has been published on this topic so far, there seems to be a systemic rot in the entirety of Boeing Engineering and Management which needs to be investigated thoroughly as a whole rather than piecemeal i.e. focus on the system itself rather than the simple cogs in the system.
Often, software engineers are pipped, written up, berated, stack ranked - and churned via various management processes for doing exactly as asked for.
Until the culture of the company aka ALL levels of management changes to full accountability without blaming the actual engineers, nothing will change. The current corporate structure just isn't set up to incentivize good work.
If it comes with the ability to say NO, or DELAYED, without management song and dance, it will get traction.
What sane person would willingly put up code that can kill others on purpose... so we're left with incompetence.
Accountability, if implemented, should be company wide.
Right now issues have passed design stage, implementation, qa... with this sort of failure you have management to blame for not putting up processes in place so its caught / hiring and keeping competent enough people.
This is sort of expected as they optimized cost to the detriment of everything else.
Finally! Thank you for stating this explicitly.
In safety critical systems specifications are everything and is always done by a team of domain experts to an enormous amount of detail (with optional formal methods verification). The actual coder has to just use the chosen implementation language carefully to meet the specification with no personal interpretations; everything has to be explicit and documented.
Never mind that this idea of an offshore team diligently implementing the spec to the letter hand-waves away the software engineering, as if it’s a mere implementation detail not intimately connected to the system delivering the desired safety and performance characteristics.
So much this. This is happening in other industries, too. Domain experts are also becoming less expert because of lack of proper feedback about their designs.
It's great to have a spec, but you should understand what's happening with the spec and how it applies to the actual code that executes.
With software we can get software that is very close to being an executable spec. By adding a layer of indirection we make everything harder.
> Never mind that this idea of an offshore team diligently implementing the spec to the letter hand-waves away the software engineering, as if it’s a mere implementation detail not intimately connected to the system delivering the desired safety and performance characteristics.
If the "offshore" team does not have the requisite Domain Expertise (which seems to be the case here), then it is Boeing's job to provide rigorous specifications and more importantly have safety checks/verification/tests/etc. in place to guarantee "correctness to spec." Problems in the specifications itself are the responsibility of the Boeing Design/Engineering team.
Do you know who is trained to be good at spec'ing software? Software engineers.
Domain Expertise is an absolute necessity to come up with Rigorous Specifications and in particular; for Safety Critical Systems.
If they can be done by a single person (whatever be his role name) all the more better but usually for Safety Critical (and highly specialized) Systems that is not possible and hence you need two (or more) people.
Ha, Ha, Ha!