

When the next leap second hits, someone somewhere will go down.Home Subscribe What is a Leap Second and why is it worth your time? 10th December 2016 on Time, Infrastructure by Christopher Demicoliīecause of irregularities in the Earth's rate of rotation, time calculated using Earth's rotation drifts away from atomic time. Last fall, with a blog post, Google described a method it calls "leap smear." Rather than add the extra second all at once, Google has modified NTP so that it adds milliseconds to clocks over a relatively long period of time. "Basically, you trick NTP, so it won't take that sudden step back, but still adds the extra second," says Marongiu.īut he calls this a "poor man's workaround." The better solution, he says, is the one used by Google. Opera's Marongiu suggests pausing a system's NTP system for a second, rather than actually moving a system's clock back. But in the meantime, others have proposed master fixes that seek to hide the sudden time changes from systems such as Linux. Some have called for an end to leap second - so that these problems can be avoided. "And then the earth stopped accelerating." "So there was a long interval when people created all sorts of new stuff and didn't have to think about that," he says. Since then, there have been leap seconds in 2005, late 2008, and now 2012. So all of the notions of cloud services and multiprocessors and so on came into existence during a period of time when leap seconds weren't happening," he says. "From 1999 to 2005, there hadn't been leap seconds. In fact, that may be part of the problem, says Steve Allen, a programmer with the Lick Observatory, just outside of San Jose, California. But, in general, the tech industry hasn't had much experience with leap seconds over the past decade and a half. "And of those two, leap seconds are the even more painful of the two."Īs Torvalds points out, synching up the earth with the time measured by atomic clocks is a tricky business. "Leap seconds and daylight savings time changes are particularly painful, though, because they have the added complexity of being ad hoc without strict rules," he says.

Developers may test for this stuff before-hand, but there's it's hard to predict how things will play out in the real world. Whenever you mess around with time, things have a pretty good chance of going wrong, Torvalds says.

But when the next leap second does come, there could be more problems. It depends on how quickly the earth spins - and that can slow down or speed up, depending on tides, weather and the flow of molten metals in the earth's core. We don't know when the next leap second will be. The Best Laid Plans of Mice and Linux Geniuses This issued is discussed on a Linux mailing list here, and it's unclear how closely this issue is tied to the hrtimer problem experienced by Reddit.
#Leap second software
On Friday, NTP began warning servers that this year's leap second was on the way, and according to Opera Software system admin Marco Marongiu, at least some Opera servers started locking up when they received the announcement. Systems such as Linux use the Network Time Protocol, or NTP, to plug into the world's atomic clocks and check the time. Other systems, however, experienced problems a day before the leap second arrived. Both Hadoop and Tomcat also depend on Linux and Java, and it would seem they were hit by the same glitch. While Reddit was struggling with its Cassandra servers, Gawker had issues with its Tomcat servers, and Mozilla had trouble with Hadoop.

#Leap second Offline
The site was all but inoperable for about 30 to 40 minutes, and it was entirely offline for about an hour and a half. From what Jason Harvey can tell, Cassandra was failing to pause Java processes, and these processes were caught in constantly spinning loops, eating up the CPU power on Reddit's servers.Įventually, Reddit solved the problem by rebooting its servers. Its servers were running an open source database known as Cassandra that was built with the Java programming language and runs atop Linux. Reddit, however, saw something a little different. Judging from Stultz's mailing list post, when the leap second hit and these hrtimers were suddenly a second ahead of the core OS, they started ringing those alarm clocks, waking up countless sleeping applications at once and overloading the machines' CPUs.
