Jailbreak
A technique of crafting prompts to bypass an AI model's safeguards and make it produce content that is normally prohibited.
In July 2023 Zou, Wang, Kolter and Fredrikson published a text suffix, found by optimization on open models, that also breaks ChatGPT, Bard and Claude. Jailbreaking covers the techniques that get around an AI model's safeguards: prompts crafted to make it produce what its designers meant to prohibit, whether dangerous instructions, illicit content or disclosures it should refuse. The methods keep changing: role-play, nested instructions, exploited ambiguities, less-monitored languages. Every vendor fix calls for a new workaround, and the cat-and-mouse game has no end point. A jailbreak shares the spirit of prompt injection, but it targets the model's internal safeguards, whereas injection hijacks an agent through external data. For insurance and risk management, it raises delicate liability questions. When a jailbroken model produces harmful content, fault is apportioned uncertainly between the user who forced the system, the vendor whose protections gave way and the company that deployed the tool without sufficient oversight.
Textbook case: a user disguises a prohibited request as a fiction exercise to get an assistant to supply instructions it would normally have refused. If those instructions cause harm, the apportionment of liability between the user, the vendor and the deployer remains largely uncertain.
Follow the research
One letter a week. First, the week's underwriting note, when one is published: a dated, argued decision, signed by Paul Bernardski. Then what the observatory published on cyber risk, political risk and insurance. No account, no tracking.
No spam, one-click unsubscribe, no data ever sold.