Every answer and its explanation appears here once you have finished the path. Each one then links to the matching glossary entry, where the concept is set out in full with its worked example.
1. Most firms in a sector hand their risk analysis to the same foundation model. A flaw in that model's reasoning produces identical, simultaneous wrong decisions across the whole sector. What has this situation destroyed?
Cognitive diversification, which naturally damped errors
Algorithmic monoculture is the software equivalent of a single point of failure, a concept pushed forward by researchers at Stanford's Center for Research on Foundation Models. In a human organization, the chance that thousands of individuals make exactly the same error at the same moment is negligible, and that dispersion of judgment acts as a shock absorber nobody had to design. A shared foundation model removes it: if the model carries an analytical bias, a flaw in reasoning or a vulnerability, the failure is no longer isolated, it is replicated at once across everyone depending on it. For an insurer the shift is large and invisible in historical data: a risk believed diversified because its carriers are independent stops being so the moment they all reason with the same tool.
Glossary entry →2. An attacker inserts trapped examples into the public data used to train a model, teaching it to misclassify one precise type of document. The model looks normal and the flaw is only exploited months later. Which attack is this?
Poisoning of the training data
Poisoning targets not the trained model but the raw material it learns from. By slipping corrupted examples into the training set, an attacker durably alters the model's behavior, either to degrade its general quality or, far more insidiously, to hide a backdoor that fires only on a signal known to them alone. Two features make it hard to counter. The first is stealth: the model behaves normally in almost every situation, so no ordinary performance test shows anything. The second is structural: learning systems depend on vast corpora often scraped from the internet, whose provenance and integrity cannot be guaranteed in practice. The difference from an adversarial example lies in when the attack lands, poisoning acting at training time where the adversarial example acts at inference.
Glossary entry →3. A motor insurer assesses damage from photographs with a vision model. A repairer submits images slightly altered by a digital filter, inflating the estimate by 40 percent with no visible tampering. What are these inputs called?
Adversarial examples
An adversarial example is an input to which small, often imperceptible perturbations have been deliberately added to force a classification or prediction error. Goodfellow and co-authors demonstrated the phenomenon in 2014 on vision networks: a correctly classified panda image can be perturbed at the pixel level and classified as a gibbon with high confidence, with nothing visible to the eye. White box attacks, where the attacker knows the model's parameters, are distinguished from black box attacks, where only outputs are available. Adversarial robustness has become a central security criterion wherever a model runs in a hostile environment, facial recognition, fraud detection, autonomous driving, insurance document processing. The case here is especially expensive because the error is systematic and one-directional: it does not average out, it accumulates.
Glossary entry →4. A researcher submits fifty thousand synthetic profiles to an insurer's instant health quote API and, by analyzing score variations, reconstructs the typical medical profiles that maximize estimated risk. What has he carried out?
A model inversion attack
Model inversion was formalized by Fredrikson and co-authors in 2015, in a paper showing that repeatedly querying a facial recognition model allowed the reconstruction of face images used in training, after a first 2014 study on a drug dosing model. The distinction from membership inference is sharp and worth holding: membership inference asks whether one specific individual was in the training set, inversion seeks to reconstruct generic representations of the private data that shaped the model. Modern techniques applied to image generators, known as GAN inversion, rebuild high resolution faces or documents by optimizing an input until it produces the target output. What matters to an insurer is that the leak passes through no database at all: it passes through a public quotation interface, that is, through a service whose very purpose is to answer repeated queries.
Glossary entry →5. By querying a health risk score API with profiles built from public court data, a researcher determines with 73 percent accuracy whether individuals with a rare disease were in the training set. What does this attack rely on?
On the model's partial memorization of training data, above all rare examples
Membership inference exploits a model's tendency to partially memorize its training data, and rare or atypical examples above all, precisely the ones that identify a person best. By querying the model on a target record and watching its outputs closely, notably the confidence attached to each class, an attacker infers well above chance whether that record was part of training. The legal point is immediate: for a model trained on medical, insurance or financial data, revealing that an individual belongs to a client book, a claims cohort or a group of people with a given condition is a potential GDPR breach. Pseudonymization does not answer it, because what leaks is not the identifier but the fact of having been present. The defense runs through differential privacy, which adds calibrated noise, at a cost in accuracy that is paid elsewhere.
Glossary entry →6. A fraud detection model trained on behavior from two years ago grows steadily less effective as fraudsters change methods. What makes this degradation particularly dangerous?
It is silent, the model going on producing normal-looking answers
Model drift is the gradual decay of a system's performance as the reality it meets in production moves away from the data it learned on. A model learns the regularities of a snapshot of the world, and the world moves: behavior evolves, distributions shift, what was true at training slowly stops being true. Data drift, where input characteristics change, is distinguished from concept drift, where the relationship between inputs and the target itself changes, and the fraud case is the second and harder kind, since the adversary adapts on purpose. The danger is the quietness: the model does not break, it raises nothing, it keeps returning plausible answers while becoming less reliable. An outright failure would cost less, because it would be seen. That is why drift demands continuous monitoring of real performance, not a check at go-live.
Glossary entry →7. A model is deprived of a protected characteristic but fed thousands of other variables. It reconstructs that characteristic's effect from its correlates, with no intent to discriminate. Why does removing the variable settle nothing?
Because banning a variable does not ban its effect, which the model recovers by correlation
Proxy discrimination is the mechanism by which a model deprived of a protected characteristic rebuilds its effect from correlates. It requires no intent, which is what makes it hard to grasp legally: a model's whole function is to find every predictive correlation, and the protected characteristic is one. ZIP code, purchase history or login times can carry the information believed removed. The consequence bites hard on non-discrimination law built around lists of prohibited variables: banning an input does not ban its effect, and formal compliance can sit alongside a discriminatory outcome. The only known remedy is to regulate outputs rather than inputs, testing for disparate impact even on an apparently neutral model. And that remedy meets a difficulty worth naming: measuring a gap along a protected characteristic requires holding that characteristic, which is exactly what collection rules forbid.
Glossary entry →8. A professional relies without checking on a legal citation fabricated by a generative AI and causes loss to their client. At the crossroads of which regimes does the question of imputation arise?
Product liability, the professional liability of whoever deployed the tool, and defamation or consumer law as the case may be
A hallucination is false, invented or misleading information produced with every appearance of reliability. It can cause perfectly concrete loss: bad advice, defamation of a person, false financial or medical information, a fabricated legal citation. The question then becomes who answers for the damage, and three candidates present themselves, the model's publisher, the professional who deployed it without adequate checking, and the end user. No single regime settles it, because the situation sits at the crossroads of three: product liability, aimed at the publisher; professional liability, aimed at whoever leans on the tool and whose duty of care did not vanish because they delegated; and, depending on the case, defamation or consumer law. Early decisions have fixed on the party who repeats the information without verifying it, which fits the logic of the duty to advise: delegating execution does not delegate responsibility.
Glossary entry →9. An analyst systematically signs off the risk assessments an AI tool produces. The day the model is wrong, the error passes and a risk that should have been declined is accepted. Which control had this bias hollowed out?
Keeping a human in the loop
Automation bias is the human tendency to place excessive trust in an automated system's outputs, to the point of relaxing vigilance and no longer checking them, including when they are wrong. It grows from two reinforcing things: convenience, since automation relieves genuine cognitive effort, and a presumption of reliability, the machine being perceived as objective and competent. Its consequence is properly paradoxical. The very control put in place to guarantee safety, keeping a human in the loop, loses most of its value once that human merely rubber-stamps. Supervision then exists on paper, it is documented, it is auditable, and it intercepts nothing. This is one of the main channels by which a model's errors spread unobstructed, and it puts in question the way many texts conceive human oversight, requiring it without ever measuring whether it is actually exercised.
Glossary entry →