Automation bias
The human tendency to place excessive trust in the outputs of an automated system, to the point of ceasing to check them, which amplifies the AI's errors.
Automation bias is the human propensity to place excessive trust in the recommendations or decisions of an automated system, to the point of relaxing vigilance and no longer checking them, even when they are wrong. It comes from two things: convenience, since automation relieves cognitive effort, and a presumption of reliability, the machine being seen as objective and competent. The consequence is paradoxical: the control meant to guarantee safety, keeping a human in the loop, loses most of its value when that human rubber-stamps the system's outputs without real scrutiny. It is one of the main mechanisms by which an AI's errors propagate without being intercepted, and it undermines the illusion of safety that formal human oversight provides. For insurance and liability law, the question is a hard one: how to attribute fault when harm results from a decision validated by a human who, in practice, no longer controlled anything? The problem will spread as decision-support tools and autonomous agents spread.
Illustration: an analyst systematically validates the risk assessments produced by an AI tool, until the day a model error slips through undetected and leads to accepting a risk that should have been declined. Oversight existed on paper, but automation bias had hollowed it out.
automation bias, automation complacency, overreliance on automation
Follow the research
One letter a week. First, the week's underwriting note, when one is published: a dated, argued decision, signed by Paul Bernardski. Then what the observatory published on cyber risk, political risk and insurance. No account, no tracking.
No spam, one-click unsubscribe, no data ever sold.