Class imbalance
A situation where the event to predict is very rare, fraud, large loss, default, which pushes a naive model to neglect it and requires suitable methods and performance measures.
In many insurance problems, the event of interest is rare: less than one percent of claims are fraudulent, a tiny fraction of policies produces a large loss, few companies default. A model trained naively on such data learns that always predicting the frequent outcome is enough to be right more than 99 percent of the time, and it becomes useless. Class imbalance calls for several responses. On the data side, rare cases can be oversampled, frequent cases undersampled, or synthetic examples generated. On the model side, errors on the rare class can be weighted more heavily. On the decision side, a flagging threshold is chosen to fit investigation capacity and the relative cost of the two types of error, rather than the 50 percent threshold. Above all, accuracy must be abandoned as a performance measure, misleading by construction, in favor of measures centered on the rare class: precision and recall, area under the precision-recall curve, detection rate among the top-scored files. Any correction of imbalance in the data distorts the calibration of probabilities, which must then be restored.
Textbook case: among 100,000 claims of which 500 are fraudulent, a model that calls everything honest scores 99.5 percent accuracy and detects no fraud. A rebalanced model, whose top 2,000 scored files are flagged, finds 300 of them: its precision is 15 percent and its recall 60 percent, the only measures that say anything useful.
Follow the research
One letter a week. First, the week's underwriting note, when one is published: a dated, argued decision, signed by Paul Bernardski. Then what the observatory published on cyber risk, political risk and insurance. No account, no tracking.
No spam, one-click unsubscribe, no data ever sold.