AI model supply chain
The set of third-party models, datasets and libraries incorporated into an AI system, most often adopted without verification.
February 2024: JFrog researchers found about a hundred malicious models on Hugging Face, nearly all in PyTorch format, whose pickle files ran code as soon as they were loaded. One of them opened a reverse shell to an outside address, and the platform's scanning had not stopped it. Few organizations train a model from scratch. They start from a published model, tune it on their own data, and add datasets and libraries pulled from public repositories. This chain mirrors open source software, with two differences that make the risk worse. A weights file cannot be read, so no code review finds anything in it. And some serialization formats allow arbitrary code to run when the model loads, which turns a download into an execution. Provenance adds another difficulty. Public model licenses vary and are sometimes incompatible with commercial use, and training data is rarely documented, so a deployer inherits exposures it cannot inventory. The remedies are those of software: a component inventory, artifact signing, and a preference for serialization formats that execute nothing. That last one is the most effective and the easiest to mandate.
In February 2024, a security research team reported around a hundred booby-trapped models on a large public model-sharing platform. Their malicious code ran on loading, through a serialization format that permits arbitrary execution.
Follow the research
One letter a week. First, the week's underwriting note, when one is published: a dated, argued decision, signed by Paul Bernardski. Then what the observatory published on cyber risk, political risk and insurance. No account, no tracking.
No spam, one-click unsubscribe, no data ever sold.