Membership inference attack

Shokri, Stronati, Song and Shmatikov, "Membership Inference Attacks Against Machine Learning Models," IEEE Symposium on Security and Privacy, 2017: the attack was run against Google's and Amazon's prediction services and against a model trained on hospital discharge records. It exploits the tendency of models to partly memorize their training data, especially rare examples and outliers. The attacker queries the model on a target record and watches its outputs closely, notably the confidence assigned to each class. From that, the attacker can tell, well above chance, whether the record was in the training set. When the model learned from medical, insurance or financial data, the attack becomes a potential GDPR violation: it reveals that an individual belongs to a customer portfolio, a disease cohort or a claims group. Defenses include differential privacy, which adds calibrated noise during training to limit memorization, gradient masking and federated model sharing. For financial regulators and data protection authorities, a model's resistance to inference attacks is becoming a component of AI compliance, separate from its predictive accuracy.