The human tendency to place excessive trust in the outputs of an automated system, to the point of ceasing to check them, which amplifies the AI's errors.
Automation bias refers to the human propensity to place excessive trust in the recommendations or decisions of an automated system, to the point of relaxing vigilance and ceasing to check them, including when those outputs are erroneous. This bias stems from a mixture of convenience, automation relieving cognitive effort, and a presumption of reliability, the machine being perceived as objective and competent. Its consequence is paradoxical, since the control device meant to guarantee safety, namely keeping a human in the loop, loses most of its value when that human merely rubber-stamps the system's outputs without real scrutiny. Automation bias is thus one of the main mechanisms by which an AI's errors propagate without being intercepted, and it calls into question the illusion of safety provided by formal human oversight. For insurance and liability law, it raises a formidable question, namely how to attribute fault when harm results from a decision validated by a human who, in practice, no longer controlled anything, a situation set to become widespread as decision-support tools and autonomous agents spread.
An analyst systematically validates the risk assessments produced by an AI tool, until the day a model error slips through undetected and leads to accepting a risk that should have been declined. Oversight existed on paper, but automation bias had hollowed it out.
automation bias, biais d'autorité de la machine, complaisance envers l'automatisation