AWS Sydney outage (June 2016)
The loss of an AWS availability zone in Sydney after a storm, when the diesel rotary UPS units drained into the failing grid instead of powering the data halls.
A data center survives a utility outage only if its backup power isolates itself from the grid fast enough to serve the data halls alone. During the storm that hit Sydney in June 2016, the loss of a regional substation did not produce a clean cut but a long voltage sag; the breakers meant to separate the site from the grid did not open in time, and the energy stored in the rotary UPS units flowed outward instead of powering the servers. Once drained, those units shut down before the generators could pick up the load, and an entire availability zone lost power. A latent software bug in the instance management tooling then slowed the return of some machines. The case shows that the backup chain is designed and tested for a clean outage, while the real grid often fails in a degraded way, and that the availability zone, the provider's unit of isolation, remains a boundary the customer must know how to reach beyond.
On June 4, 2016, at 10:25 p.m. Pacific time, Sunday afternoon June 5 in Sydney, the loss of a regional substation during a violent storm cut power to several AWS facilities. According to Amazon's account as reported by the press, the diesel rotary UPS units in one availability zone drained into the grid, taking down EC2 and EBS in that zone and hitting customers such as Foxtel Play, Stan and Channel Nine, until full recovery on Monday morning.
Follow the research
One letter a week. First, the week's underwriting note, when one is published: a dated, argued decision, signed by Paul Bernardski. Then what the observatory published on cyber risk, political risk and insurance. No account, no tracking.
No spam, one-click unsubscribe, no data ever sold.