An attack that deliberately corrupts a model's training data to bias its behavior or to insert a backdoor.
Data poisoning is an attack that targets not the already-trained model but the raw material it learns from, namely its training data. By surreptitiously introducing corrupted examples into the learning set, an attacker can durably alter the model's behavior, either to degrade its quality generally or, more insidiously, to conceal within it a backdoor that will trigger only in response to a precise signal known only to the attacker. This threat is particularly worrying because it is hard to detect, the model appearing to work normally in most situations, and because it exploits AI systems' dependence on vast corpora of data often gathered from the internet, whose provenance and integrity cannot be fully guaranteed. Poisoning thus shifts the attack surface upstream in the AI production chain, making it a particular case of supply chain risk applied to models. For insurance, it illustrates a new category of potential loss, latent and deferred, in which the damage is sown well before it manifests, which considerably complicates dating the triggering event and attaching it to a policy period.
An attacker inserts, into the public data used to train a model, booby-trapped examples that teach it to misclassify a specific type of document. The flaw remains invisible until the attacker, months later, exploits it against the deployed model.
data poisoning, empoisonnement de données, corruption des données d'entraînement