Overfitting
The flaw of a model that has learned the noise in its training data instead of the patterns, brilliant on the past and mediocre on new risks, a permanent danger of flexible models on sparse claims data.
An overfitted model reproduces too faithfully the data it was trained on, including their accidents and coincidences, to the point of losing its ability to predict new cases correctly. It shows excellent results on the training sample and markedly worse results on an independent sample. The phenomenon is all the more likely when the model is flexible, deep trees, neural networks, many variables, and the data are few or noisy, which is precisely the case with large claims, rare by definition. In insurance it has direct consequences: an overfitted tariff assigns very different premiums to policyholders with identical risk, simply because chance struck some and not others in the history, and it degrades as soon as it goes live. The remedies are known: strict separation of training, validation and test data, cross-validation, regularization that penalizes complexity, limiting tree depth, and, in actuarial science, credibility, which blends a segment's own experience with that of the portfolio in proportion to its statistical reliability.
Textbook case: a 400-variable model explains 92 percent of cost variance on training claims, but only 35 percent on the following year. Cut to 40 variables and regularized, it explains only 60 percent on training, but 55 percent on the following year: the second, less brilliant model is the only useful one.
Follow the research
One letter a week. First, the week's underwriting note, when one is published: a dated, argued decision, signed by Paul Bernardski. Then what the observatory published on cyber risk, political risk and insurance. No account, no tracking.
No spam, one-click unsubscribe, no data ever sold.