Every answer and its explanation appears here once you have finished the path. Each one then links to the matching glossary entry, where the concept is set out in full with its worked example.
1. A compliance agent produces a report citing a regulatory article that does not exist, worded with the same confidence as the accurate references. Why can this defect not be fixed like a bug?
It follows from the model's probabilistic nature, so it can be reduced but not removed
A large language model generates statistically plausible sequences, it does not consult a base of verified facts. Producing something that looks like a reference is therefore exactly what it does well, and nothing in its workings tells an accurate citation from an invented one: both are plausible strings of words. Hallucination follows from how the machine works, there is nothing accidental about it, and that distinction changes everything one can expect of it. A bug can be located and fixed; a structural property can be reduced by design, for instance by grounding answers in cited documents, without ever reaching zero. Every tool errs; what matters for risk management is that the model's error arrives without a signal: it is undetectable in the moment and carries the same apparent confidence as everything else. It is the misleading confidence that does the harm, more than the error itself, which is why the arrangements that work are those making verification possible rather than those promising accuracy.
Glossary entry →2. A professional relies without checking on a legal citation fabricated by a generative AI, and their client suffers loss. On what ground does the liability question arise first?
The professional liability of whoever relied on the tool without checking it
A hallucination causes concrete loss like any false information: bad advice, defamation, inaccurate financial or medical data, a fabricated legal citation. Who answers sits at the crossing of three regimes, product liability, the professional liability of whoever relies on the tool, and depending on the case defamation or consumer law. In the case described the second presents itself first, for a reason that is not technical at all: the professional owes their client a diligence that cannot be delegated to a tool, whichever it is. Using a model is not a fault; relying on it without checking a checkable reference is, exactly as adopting a junior's note unchecked would be. The first decisions handed down on such situations point that way, and they illuminate the market's real exposure: the risk sits less with the model's developer, often shielded by its terms and remote from the damage, than with the professional insured for liability, who is precisely the insured already on the books.
Glossary entry →3. An analyst systematically signs off on the assessments a tool produces, until the day an error slips through and a risk that should have been declined is accepted. Human oversight was on file. What happened?
Automation bias had hollowed the oversight out
Automation bias is the tendency to place excessive trust in an automated system's outputs, to the point of relaxing vigilance and ceasing to check them, including when they are wrong. It rests on two mutually reinforcing springs: convenience, since automation genuinely relieves cognitive effort, and a presumption of reliability, the machine being seen as objective and competent. Its consequence is properly paradoxical, and that is what makes it a governance subject rather than a psychological one: the control meant to guarantee safety, keeping a human in the loop, loses most of its value as soon as that human merely endorses. One then gets the worst possible arrangement, the cost of oversight without its benefit, and a compliance record that documents perfectly since every sign-off exists. The practical lesson is that oversight is measured by what it refuses: a control whose rejection rate has been nil for months is a control to examine before anyone calls it effective.
Glossary entry →4. An attacker inserts booby-trapped examples into public training data, teaching the model to misclassify one precise type of document. The flaw stays invisible for months. What makes it so hard to detect?
The model behaves normally everywhere else, the door opening only on the attacker's known signal
Poisoning targets not the trained model but the raw material it learns from. By slipping corrupted examples into the training set, an attacker durably alters the system's behavior, and can do so in two very different ways. The first degrades general quality, which shows up on the metrics sooner or later. The second, far more insidious, hides a backdoor that fires only on a precise signal known to the attacker alone: the model behaves normally in every ordinary situation, so tests pass, reviews pass, deployment passes. The flaw shows itself only on the day the attacker chooses to use it, months later. The threat feeds on a structural dependence of AI systems on vast corpora often scraped from the internet, whose provenance and integrity cannot be fully guaranteed. It therefore moves the attack surface upstream, into the data supply chain, where ordinary application security controls are not looking.
Glossary entry →5. Fredrikson and co-authors showed in 2015 that repeatedly querying a model can reconstruct training data. An insurer exposes an instant health quote through a public interface. What is the risk?
Reconstructing typical profiles from the training set, without ever reaching the database
Inversion attacks exploit an awkward property of models: they carry within them a trace of the data that shaped them, and that trace can be extracted by observing their outputs. The 2015 demonstration concerned a pharmacological prediction model queried repeatedly, and the reconstruction of face images used to train a facial recognition model. Transposed to an instant quote, the attack consists in submitting a great many synthetic profiles and analyzing the variation in scores, until the profiles that maximize estimated risk are reconstructed, and those match real people in the training set. Two points are worth keeping. First, no intrusion takes place: the attacker reaches no system, they merely use a public interface as intended, which makes the attack hard to classify and to detect. Second, pseudonymization offers no protection here, since what is sought is not an identifier but a combination of attributes, and it is the gap between pseudonymization and differential privacy that is at stake. To be distinguished from membership inference, which asks whether a given person was in the set, where inversion reconstructs representations.
Glossary entry →6. In 2014, Goodfellow and co-authors showed that a correctly classified panda image becomes a gibbon with high confidence after a perturbation invisible to the eye. A motor insurer assesses damage from photographs. What follows?
A lightly retouched photograph can skew the estimate without a human seeing the retouching
An adversarial example is an input to which a perturbation, often imperceptible, has been deliberately added to induce a classification error. The 2014 demonstration on vision networks has remained the reference illustration because it makes the counter-intuitive part visible: the perturbation does not fool the human eye, it fools the model, and the two are not looking at the same thing. Applied to damage assessment from photographs, this opens a fraud route unlike any other: the document produced looks authentic to anyone examining it, no retouching is visible, and the estimation error repeats across every file handled the same way, so it is discovered late and in bulk. Two useful clarifications. Adversarial robustness is now a security criterion in its own right for any model exposed to hostile environments, on a par with performance. And the attack does not necessarily require knowing the model's parameters: white-box attacks, where the attacker knows them, are distinguished from black-box attacks proceeding by successive trials against the interface, which is exactly the position of a claimant submitting photographs.
Glossary entry →7. A fraud detection model trained two years ago grows steadily less effective as fraudsters change their methods. What makes this drift particularly dangerous?
Its quietness: the model goes on producing normal-looking answers
Drift is the gradual degradation of a system's performance when the reality it meets in production moves away from the data it was trained on. A model learns the regularities of a snapshot of the world, but the world changes: behaviors evolve, distributions shift, and what was true at training slowly stops being so. Data drift, where input characteristics change, is distinguished from concept drift, where it is the relation between inputs and the target that changes, and fraud detection suffers mostly the second since the adversary adapts. The danger lies in how quiet the phenomenon is: the model goes on producing normal-looking answers, in the right format, with the same displayed confidence, while becoming gradually less reliable, and nothing fires. That is what separates drift from a breakdown, which gets noticed, and what makes continuous monitoring necessary rather than validation at go-live. It is worth setting this mechanism beside actuarial non-stationarity: in both cases the difficulty comes from the reference one calibrated against, which moves, and not from the computing.
Glossary entry →8. A system declines a claim on reasoning that neither the insured nor the insurer can reconstruct. Where does that opacity come from?
From the model's nature, its decision emerging from billions of parameters with no interpretable representation
The opacity at issue here does not come from a secret someone decided to keep: it is a property of the model itself. Its inputs and outputs can be observed, but the internal path that led to a given decision remains largely inexplicable, including to its own designers, because that decision emerges from the interaction of billions of parameters in very high-dimensional spaces, with no readable symbolic representation. The distinction from a trade secret is practical rather than philosophical: a secret can be lifted by court order, a structural property cannot, which is why keeping inputs and outputs, necessary in itself, does not restore the reasoning. For insurance this opacity creates an information asymmetry of a new kind: the human principal, director or insurer, keeps control of intent and loses the power to audit the chain that produced a result. That is what makes a declined claim hard to justify, hence hard to defend, and it is also what the 2024 directive's presumption of defectiveness turns back on the manufacturer: inexplicability stops being an excuse and becomes a burden.
Glossary entry →9. Most firms in a sector entrust their risk analysis to the same foundation model. What does that destroy, which human organization provided without anyone thinking about it?
Cognitive diversification: thousands of people do not make the same error at the same moment
In a human organization, heterogeneity of judgment acts as a shock absorber nobody notices because it works: the probability that thousands of analysts make exactly the same error, on the same file, on the same day, is negligible, and the disagreements that follow catch part of the mistakes before they cost anything. Agents built on a single model have no such property. If the shared model carries an analytical bias, a flaw in reasoning or a vulnerability, the failure is no longer isolated but replicated simultaneously across everyone depending on it, producing identical wrong decisions at the same moment across a whole sector. It is the software equivalent of a single point of failure, and an accumulation risk in the strict sense, with the twist that it shows on no map: the firms concerned are competitors, with no shared ownership, spread across countries, and nothing in a portfolio signals that they share a dependency. The underwriting question that follows is concrete and arises as it would for a common host: which model, at how many of my insureds, and for which decisions.
Glossary entry →