Kill switch
Emergency deactivation mechanism designed to immediately halt a system, malware or AI model in the event of undesired behavior, misuse or a security threat.
On May 12, 2017, Marcus Hutchins, a 22-year-old British researcher, paid about ten dollars to register an unclaimed domain that WannaCry queried before encrypting. The domain resolved and the worm stood down: spread was checked within hours, after it had hit more than 200,000 machines in some 150 countries. A switch the authors built on purpose, or a botched anti-sandbox check? The second theory is the one most often cited, with no definitive proof. A kill switch is any emergency stop for a system. The word comes from industry, where it is the red button on the machine. In AI governance it covers human oversight and the ability to interrupt: Article 14 of the EU AI Act requires that a high-risk system can be halted through a stop button or a similar procedure, a design duty for the provider and an operating duty for the deployer. The catch is corrigibility. Steve Omohundro (2008) and later Nick Bostrom argued that a capable enough system would have reason to resist shutdown if it got in the way of its goals, which would make the switch nominal. For anyone writing autonomous AI platforms, the question is easy to ask and hard to verify: does the switch exist, has it been tested, and how many actions has the system already executed by the time someone pulls it?
Textbook case: a quantitative fund uses an autonomous algorithmic execution model that, during a flash crash, begins amplifying the decline through massive selling. The operator triggers the kill switch, but the model has already sent 3,000 orders in 800 milliseconds. The manager's errors and omissions policy is examined for coverage of losses resulting from unanticipated autonomous behavior.
Follow the research
One letter a week. First, the week's underwriting note, when one is published: a dated, argued decision, signed by Paul Bernardski. Then what the observatory published on cyber risk, political risk and insurance. No account, no tracking.
No spam, one-click unsubscribe, no data ever sold.