configuration files strike again - remember knight capital?

Also, does anyone have a link to statistics on global BGP software usage? I'm curious what the marketshare looks like.

Perhaps the progressive rollout should wait for an affirmative conclusion instead of assuming no news is good news? I'm not being snarky, there may be some reason they don't do this.

PS. To the downvoters, truth hurts.

All this could have been contained if they deployed changes on different regions at different times. That would also help with screwing less your overseas users by running a maintenance at 10am their local time :-)

"In other words, they simply didn't test their (global!) custom route announcement management software. An edge case was triggered in production, and unsurprisingly they automatically went offline."

That said, this is the second major asia-east1 downtime in 90 days:

https://status.cloud.google.com/incident/compute/16002

It seems obvious to me that the push system should not proceed without confirmation from the management software, and the management software should not confirm the change is OK if it detects failure.

I see a straightforward defect here, not a confluence of edge cases.

> These safeguards include a canary step where the configuration is deployed at a single site and that site is verified to still be working correctly, and a progressive rollout which makes changes to only a fraction of sites at a time, so that a novel failure can be caught at an early stage before it becomes widespread. In this event, the canary step correctly identified that the new configuration was unsafe. Crucially however, a second software bug in the management software did not propagate the canary step’s conclusion back to the push process, and thus the push system concluded that the new configuration was valid and began its progressive rollout.

Taking no cofirmation of the canary testing process as a signal to go ahead though is not just a bug but a design flaw IMO.

> In this event, the canary step correctly identified that the new configuration was unsafe. Crucially however, a second software bug in the management software did not propagate the canary step’s conclusion back to the push process, and thus the push system concluded that the new configuration was valid and began its progressive rollout.

The second one is covered in the article, their system for that purpose crashed and then the system that babysits that crashed and then whatever they use to monitor the monitors monitor system didn't notice. Probably showed up in some dude's nightly syslog file dump the next day. Oh well. If your monitor tool breaks due to complexity (as they often do) it needs to simplicate and add lightness not slather more complexity on. Usually monitoring is more complicated and less reliable than operating, its harder computationally and procedurally to decide right from wrong than to just do it.

The odds of cascaded failure happening are very low. Given fancy enough backup systems that means all problems will be weird cascaded failure modes. That might be useful in training.

When I was doing this kind of stuff I was doing higher level support so see above at least some of my stories are weird cascaded impossible etc. A slower rollout would have saved them, working all by myself I like to think I could have figured it out by comparing BGP looking glass results and traceroute outputs from multiple very slowly arriving latency reports to router configs with papers all over my desk and multiple monitors in at most maybe two days. Huh, its almost like anycast isn't working at more sites every couple hours, huh. Of course their automated deployment is complete in only 4 hours, which means all problems that take "your average dude" more than 4 hours of BAU time to fix are going to totally explode the system and result in headlines instead of a weird bump on a graph somewhere. Given that computers are infinitely patient, slowing down the rollout of automated deployments from 4 hours to 4 days would have saved them for sure. Don't forget that normal troubleshooting places will blow the first couple hours on written procedures and scripts because honestly most of the time those DO work. So my ability to figure it out all by myself in 24 hours is useless if the time from escalation to hit the fan was only an hour because they roll out so fast. Once it hit the fan a total company effort fixed it a lot faster than I could have fixed it as an individual.

Or the strategy I proposed where computers are also infinitely fast, roll out in five minutes, one minutes to say WTF, five minutes to roll back, 11 minute outage is better than what they actually got. Its not like google is hurting for computational power. Or money.

I'm sure there are valid justifications for the awkward four hour rollout thats both too fast and too slow. I have no idea what they are, but the google guys probably put some time into thinking about it.

[1] https://plus.google.com/+RipRowan/posts/eVeouesvaVX

Really? That sounds counter intuitive. You'd think the reason people are afraid of flying is, because, you know, it's flying. Thirty thousand feet between you and the cold, hard ground. That's a long fall of agony to utmost certain death, and some magic turbo voodoo keeping you from it.

Would people with fear of flying really rather be the pilot?

In a failure case, it should remove the failing config, not all of them.

Pretty hard thing to miss if you test for it with any level of basic unit test or similar.

Second bug: canary failure should prevent further propogation of the bad config.

A little more difficult to test with automated tests due to requiring a connection. It sounds like this was in fact tested, but the usage between the two bits of software was not tested. A good integration test would have caught this. But I wouldn't call that required. I would at least however think it was required that the use case of that particular code to be at least manually checked because, you know it's a feature for disaster prevention / recovery.

There was enough information to deduce this pretty easily. Although they did tend to glaze over it in the write-up, almost purposefully.

For all those spouting that this was a good postmortem, not really, it's a good covering of ones ass, a good spin, sidestepping the real root cause.

What has slas and "here take credits" got to do with a postmortem?

I'm not really sure why I got downvoted for this. The post mortem was good but it wasn't something I'd aim to strive for. I like gcloud and I'll keep using it but I find the response to this thing a little bit hard to swallow.

Its turtles all the way down!

As for cars, it's a real risk, but not the same as the bugs Google experienced; I personally have experienced a "bug" driving a car at high speeds, which resulted in a number of major electronic systems failing due to custom systems installed by a well known US startup.

A stroke or heart attack while driving?

For self-driving cars to be safer than human drivers, there is no requirement that the self-driving cars should be better/safer than the best human driver... the self-driving car simply needs to be safer than the majority of humans.

Here's Google talking about it: https://static.googleusercontent.com/media/www.google.com/en...

Of course this is only one component to feeling confident about pushing out an update where lives are on the line, but it's a key component.

I thought programmers were on here :-)

I don't doubt that Google's infrastructure is as complicated and nuanced as it can get. Configuration software just simply isn't.

You literally have no idea what you're talking about.

In other words, I agree that it's better for society, but that "better for society" isn't the metric that gets used for making decisions within the system.

I'm sorry but the important point is: how are we going to agree what needs more testing, and what can be updated without testing? If we let those big companies decide about those issues, then I'm afraid we will soon see another scandal like VW, except possibly with deadly consequences.

I can already predict the reasoning of those companies: last quarter our cars were safer than average, so we can afford some failures now.

All pipe dreams of mine, but the research potential here could be worth flaming truckloads of grant money :-)

Top Gear discussed speed limits being set based on worst-case breaking distances. Here's an analysis of that: http://www.jmp.co.uk/forward-thinking/update/top-gear-and-sp... but TL;DR performance vehicles can be safer since they're more capable.

Think what an F1 or rally car could do with computers driving it and avoiding accidents.

It's not possible to have a self-driving car avoid all accidents, but one can presumably get pretty close. The realtime data from sensors give you enough information about the car itself, its surroundings and other objects around it to continuously compute a safety envelope - a subspace of the phase space of controllable parameters (like input, steering) within which the car can stop safely - and then make one of the goals to aggressively steer the car to remain in that envelope. This approach should be able to automagically handle things like safe driving distances or pedestrians suddenly running into the street.

Of course there will be a lot of details to account for when implementing this software, but it's important to realize that we have enough computing power to let the car continuously have every possible backup plan for almost any contingency in its electronic brain.

I hardly think the argument will be difficult to just prohibit cars.

First, they're not anywhere near 100% reliable. They can fail on their own, and they'll also intentionally shut themselves off if the instruments they rely on fail. https://en.wikipedia.org/wiki/Air_France_Flight_447

Second, an autopilot failure shouldn't lead to death if the pilots are competent and paying attention.

Actually it is much more simpler than a self-driving car. And if there is a problem it disengages.

But even this relatively simple level of automation causes problems - pilots start to rely on automation too much, and when things go south they are not capable to deal with it.

Airlines recognize it, and put more emphasis on hand-flying during training and routine operations, so pilots don't lose their basic piloting skills.

It's not a new problem - there is an excellent training video from 1997 - "Children of the Magenta": https://www.youtube.com/watch?v=pN41LvuSz10

It's 2025 and more than 10% of the cars on the road in the US are self-driving. It's rush hour on a busy Friday afternoon in Washington, DC. Earlier that day, there'd been a handful of odd reports of self-driving Edsels (so as not to impugn an actual model) going haywire, and the NTSB has started its investigation.

But then, at 430pm, highway patrol units around the DC beltway notice three separate multi-Edsel phalanxes, drivers obviously trapped inside, each phalanx moving towards the Clara Barton Parkway, which enters DC from the west. Other units notice four more phalanxes, one comprising 20 Edsels, driving into DC from the east side, on Pennsylvania Avenue.

At this point, traffic helicopters see similar car clusters, more than two dozen, all over DC, all converging on a spot that looks to be between the Washington Monument and the White House.

We zoom in on the headquarters of the White House Secret Service. A woman is arguing vociferously that these cars have to be stopped before they get any closer to the White House. A colleague yells back that his wife is one of those commandeered cars and she, like the rest of the "hackjacked" drivers and passengers is innocent.

There are a few instances where some bad tesla batteries (the standard 12 volt batteries ironically) failed, and the cars handled it perfectly. It slowed down so that the driver could do safely pull over. Sure it did not happen to all cars at once, and autonomous cars migh not be able to do that by themselves but we have a log way to go to reach 100% autonomous driving (i.e. Without a steering wheel and a car that drives everywhere humans drive and not only on San Francisco's perfect sunny roads where it's been thoroughly tested on).

Pilots were trained on the procedure for recovering from a low altitude stall; 100% or TOGA thrust and power out of it while minimizing altitude loss. Training has now changed for both low altitude and high altitude stall recovery.

Gratuitous CGI?

Will Smith saying "Awww, hell naw"?

https://www.youtube.com/watch?v=L1UxZJ9owXY

Auto mfgs seem to be about 20-30 years behind when it comes to computers. Not really surprising that Tesla is whomping them on this front, given how SV people are scrambling to work there. You don't see that with the Big 3 or really any other car mfg.

Of course a competent writer would've thrown in a line about how these cars are on run flats at some point...

Our only hope is for the scientists in the So-Secret-President-Doesnt-Even-Know Facility to come up with something so crazy it just might work

This gives us the classic reassuring response from Boeing spokeswoman Lori Gunter :

"There are places where the networks are not touching, and there are places where they are," she said.

http://www.wired.com/2008/01/dreamliner-security/

Airplaine components tend to have shitloads of fuses for each components, any trained pilot knows how to disable the fuse for the autopilot system (or, in an extreme case, ALL fuses to kill the entire airplane).

Another reason traffic spoofing wouldn't cause the aircraft to deviate is that airliners fly standard approaches and departures (STAR [1] and SID [2]) and heavy traffic away from the approach paths would definitely get noticed.

Even the fly-by-wire Airbus can be flown manually using differential thrust and/or pitch trim control.

The only time I've heard of an Airbus loosing control of a damaged engine is when the electrical cable was physically severed. This was Qantas QF32 [1], after one engine exploded and damaged the cables to another engine.

To "take over" an aircraft with pilots in the cockpit, would require the compromise to multiple systems.

[0] https://en.wikipedia.org/wiki/Traffic_collision_avoidance_sy...

[1] https://en.wikipedia.org/wiki/Standard_terminal_arrival_rout...

[2] https://en.wikipedia.org/wiki/Standard_instrument_departure_...

[3] https://en.wikipedia.org/wiki/Qantas_Flight_32

Google cars have the Big Red Button, which shuts off self-driving system and brings the car to a stop.

What more controls do you need?

It won't matter if all the other cars on the road besides yours don't have controls.

>In some cases, people are having to wait months to get new airbags because they just don't have them in stock. In the computer case, would you want to keep driving until they can get you scheduled for a software update? Remember that many cars can't update critical software OTA.*

So I assume you don't own a car and you avoid them at all costs? Otherwise your paranoia becomes hypocrisy. If you cannot trust the car company to deliver software updates, you can't trust them to write the software in the first place, and modern cars are full of safety-critical software.

I also don't know why you're equating the a manufacturing capacity limitation with a software update limitation. It's not as if Toyota is going to have trouble shipping bits a million times vs a thousand times once the software update is written.

I think we can also safely assume that self-driving cars will generally be updatable OTA. But yes, you could drive it to the dealer if needed, and worst case the dealer could send people on-site to do the update.

I say allegedly, because my local Honda dealership told me to pound sand when I asked for a rental car for the day they needed to repair my CRV.

There is an idea of triple channel autolanding, wherein the plane uses the consensus of the three autolanding systems. Should no consensus be available, then the pilot is advised that autolanding is not available.

Other than that, any sourcing from different manufacturers is happenstance. 737 avionics are sourced from a different vendor than 747/757/767/777. And different functions can come from different vendors, although vendor consolidation has cut down on that.

I'm not across what happened post 777, as I left Boeing in 1999.