Cross-validation
An evaluation method that splits the data into folds, trains the model on all but one and tests it on that one, in turn, to measure its performance on unseen data without sacrificing a sample.
To know whether a model will predict new cases well, it must be evaluated on data it did not see during training. Setting aside a test sample is the simplest solution, but it wastes precious data when claims are few, and the result depends on the chance of the split. Cross-validation divides the data into k equal folds, often five or ten, trains the model k times, each time on all folds but one, and measures its performance on the fold left out. The average of the k measures gives a more stable estimate of real performance, and their dispersion indicates its reliability. The method also serves to choose a model's settings, tree depth, regularization strength, without touching the final test sample. In insurance two precautions are essential. The data have a time dimension: a model validated by mixing years may look better than it will be on the future, and validation must often be done on the most recent years. And observations from the same insured or the same event must stay in the same fold, otherwise the model is tested on what it has already seen.
Textbook case: a team validates a frequency model with five-fold cross-validation mixing all years and obtains a Gini of 0.40. Validating instead on the last year, after training on the earlier ones, it obtains only 0.33: the second, stricter measure is the one that predicts what the tariff will actually do.
Follow the research
One letter a week. First, the week's underwriting note, when one is published: a dated, argued decision, signed by Paul Bernardski. Then what the observatory published on cyber risk, political risk and insurance. No account, no tracking.
No spam, one-click unsubscribe, no data ever sold.