Model extraction attack
Reconstructing a proprietary model by repeatedly querying its interface, transferring value without crossing any security perimeter.
Tramèr, Zhang, Juels, Reiter and Ristenpart, "Stealing Machine Learning Models via Prediction APIs," USENIX Security, 2016: models hosted by BigML and Amazon were rebuilt with a few hundred to a few thousand queries. The attacker queries a prediction interface a very large number of times with chosen inputs. They then train a local model on the resulting input and response pairs until they hold a substitute of comparable performance. The attack requires no intrusion, no stolen credentials and no software vulnerability: it uses only the service as sold. Security tools do not see it, and the law struggles to characterize it. The harm is nonetheless real, and twofold. The value of the training investment changes hands, and the substitute then serves to prepare further attacks, because adversarial examples built against it often transfer to the original model. The usual countermeasures all trade protection against service quality: query rate limiting, less precise returned scores, detection of systematic querying patterns, model watermarking. For insurance, this exposure belongs to intellectual property rather than to a conventional cyber policy, and many wordings do not yet make the distinction.
The paper Stealing Machine Learning Models via Prediction APIs, presented at USENIX Security in 2016, showed that a commercial model exposed through a prediction interface could be reconstructed with high fidelity from a limited number of queries.
Follow the research
One letter a week. First, the week's underwriting note, when one is published: a dated, argued decision, signed by Paul Bernardski. Then what the observatory published on cyber risk, political risk and insurance. No account, no tracking.
No spam, one-click unsubscribe, no data ever sold.