Model drift
The gradual degradation of a model's performance as reality diverges from the data on which it was trained.
Zillow shut down Zillow Offers on November 2, 2021, after a $304 million inventory write-down in the third quarter. The company bought homes at the price its model estimated, and management put the failure down to the difficulty of forecasting prices. Model drift is the gradual degradation of an artificial intelligence system's performance when the reality it meets in production moves away from its training data. A model learns the regularities of a snapshot of the world. The world changes, behaviors evolve, distributions shift, and what was true at training gradually stops being so. Practitioners often separate data drift, when the characteristics of the inputs change, from concept drift, when it is the relationship between the inputs and the target that changes. The danger is its stealth: the model keeps producing normal-looking answers and grows less reliable with no obvious alert. Performance has to be monitored continuously and the model retrained at regular intervals. In cyber the issue is acute. Attackers keep adapting their techniques, and detection models drift fast, to the point of going blind to new kinds of attack. Drift is thus an operational and liability risk, since an unmonitored system can cause harm long after it has stopped being reliable.
Textbook case: a fraud-detection model trained on behaviors from two years ago becomes gradually less effective as fraudsters change their methods. For lack of monitoring, its drift goes unnoticed until a wave of frauds slips past it.
Follow the research
One letter a week. First, the week's underwriting note, when one is published: a dated, argued decision, signed by Paul Bernardski. Then what the observatory published on cyber risk, political risk and insurance. No account, no tracking.
No spam, one-click unsubscribe, no data ever sold.