Internal network failure (hyperscaler)
Outage caused by a configuration error in a major operator's own internal network, able to cut access to its services with no hardware failure at all.
Internal network failure at a hyperscaler is an outage that comes from a configuration error in the operator's internal logical network, in particular routing protocols such as BGP, rather than from physical damage. These protocols govern how the parts of a global infrastructure communicate with each other. A poorly validated configuration change can cut access to an operator's services worldwide, even as every one of its data centers remains physically intact and powered. This peril comes from the nature of very large digital infrastructures. Their scale and automation normally make them resilient to local hardware failures, but they expose them to software failures that spread across the whole system within minutes, with no guarantee that immediate manual intervention can stop them. Recovery can also drag if the remote management tools depend on the network that has gone down, forcing engineers to intervene physically on site. For insurers, this peril reinforces the accumulation risk already identified for hyperscalers: a purely software event can reach as far as a physical disaster.
On October 4, 2021, a configuration error in Facebook's internal routing network cut off global access to Facebook, Instagram and WhatsApp for about six hours, and even prevented some employees from physically entering data centers whose badge systems depended on the same network.
Follow the research
One letter a week. First, the week's underwriting note, when one is published: a dated, argued decision, signed by Paul Bernardski. Then what the observatory published on cyber risk, political risk and insurance. No account, no tracking.
No spam, one-click unsubscribe, no data ever sold.