Tell HN: Both recent GCP outages caused by fiber optic maintenance

19 points by fastest963 3 days ago | 1 comment

Both of GCP's recent outages were caused by "fiber optic maintenance". On August 20th when us-west1 went down they said [1]:

> The disruption originated during scheduled fiber optic maintenance, which unexpectedly compromised network capacity between data centers within the us-west1 region.

Then on September 1st when us-central1-b (and collateral affects in other zones) went down they said [2]:

> The immediate technical trigger for this event was the inadvertent physical disconnection of network fiber-optic cables during a routine hardware maintenance procedure.

Not sure if they're related but apparently procedures we're significantly changed after the first incident to prevent the second incident.

[1] https://status.cloud.google.com/incidents/utF3FMFdQfwBzJcGG6vf

[2] https://status.cloud.google.com/incidents/J5ia5t9p3g9Q5Wi7r8Ev

mjuarez 3 days ago |

> Not sure if they're related but apparently procedures we're significantly changed after the first incident to prevent the second incident.

Which obviously didn't work. Or worse, made the second incident more likely.