How a leap second caused havoc with the Internet (2021)

How a leap second caused havoc with the Internet (2021)(talesfromtheopsside.com)

40 points by frantic2821 2 years ago | 12 comments

nytesky 2 years ago |

A couple years ago we had issues caused by a 0 (zero) leap second, which honestly I couldn’t find documented anywhere but hacker news! Something about the system expects a leap second every 17 years and if it doesn’t get one it’s more troublesome than the 0 leap second.

https://news.ycombinator.com/item?id=27944776

gertrunde 2 years ago |

One of the more unseen side-effects of this particular bug back in 2012 was datacenter internal temperatures spiking by ~5℃ due to the increased CPU usage iirc.

ahazred8ta 2 years ago |

We've already discussed the Leap Smear, where clocks run 0.0012% slower for a day.

https://news.ycombinator.com/item?id=28047714

https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

zokier 2 years ago | |

And inevitably there is also Leap Unsmear: https://news.ycombinator.com/item?id=22223591

By Google. Who started this whole trend of leap smearing in the first place.

tinix 2 years ago | | |

Google didn't start this... It's standard practice to speed up or slow down the clock during transition in ERP systems.

VAX/VMS are way older than Google.

ahazred8ta 2 years ago | | |

Actually, before 1972 they would run the radio time signals several parts per million fast or slow to keep in step with the earth's rotation.

gregw2 2 years ago |

I recall this hitting our startup’s various MySQL Linux servers in 2012.

They were slow/maxed out for a while till we figured it out. The DBA rebooted a few but didn’t know why and it kept cropping up on others; I drilled into root cause with some google-fu.

Not mentioned in the article is that I think this issue occurred again in 2015. I don’t recall if it hit us because we hadn’t patched/upgraded some servers since 2012 (!) and had just relied on resetting the clock/rebooting, or there were more Linux bugs (see LWN below).

Good technical writeup from Linux weekly news: https://lwn.net/Articles/504744/

And more on Linux patching in 2015: https://lwn.net/Articles/648313/

IANA canonical list of leap seconds: https://data.iana.org/time-zones/tzdb-2018a/leap-seconds.lis...

ooterness 2 years ago |

> Hany explains: “Due to physics, at certain times of the year, the Earth moves faster along its orbit than others. How do we base time off of this, then?”

Sorry, but no. This is so wrong I don't know where to start.

zokier 2 years ago | |

It's not wrong really, Earth's orbital velocity does indeed vary. It just doesn't have anything to do with leap seconds

frantic2821 2 years ago |

TLDR: a leap second overloaded computer systems due to issues in how Linux handles time.

ripe 2 years ago | |

Time to plug my old explanation of leap seconds:

https://robotsinplainenglish.com/e/2022-11-20-stopwatch-time...

zokier 2 years ago | | |

> As you can see, UTC is not a proper, continuous time stream. It has breaks in it wherever leap seconds have been added

I can not see that. 23:59:58 - 23:59:59 - 23:59:60 - 00:00:00 is perfectly continuous, proper, and even monotonic, or at least as much as 23:59:58 - 23:59:59 - 00:00:00 is.

> When you receive a package with a computer-generated time stamp, it is showing this fudged UTC number, not the actual number of seconds according to the atomic stream.

I think you are mixing up UNIX timestamps (which are their own thing) and UTC. UTC and TAI both indicate exactly same thing, there is perfect 1:1 mapping between UTC timestamps and TAI timestamps (ignoring pre-1972 era).