Champion-challenger approach
A practice that pits the incumbent model, the champion, against one or more candidate models under real conditions, on a fraction of business, before deciding which to roll out, so as to judge on observed results rather than backtests.
A model that shines in backtesting can disappoint in production: real data differ from history, customer and broker behavior reacts to the new price, selection effects appear. The champion-challenger approach addresses this risk by running the incumbent model alongside one or more challengers, each applied to a random fraction of business or files, for long enough to measure their real effects: hit ratio, losses, profitability, fraud detection rate, satisfaction. The challenger replaces the champion only if it shows a significant advantage. The practice comes from credit and marketing; in insurance it applies to claims triage, fraud detection, follow-up strategies and, with more caution, pricing. It raises sector-specific questions: losses emerge slowly, which lengthens tests; applying two different tariffs to comparable customers raises a fairness and sometimes a regulatory compliance question; and the fraction given to the challenger must be large enough to conclude, small enough to limit risk if the challenger turns out badly.
Textbook case: an insurer gives 10 percent of its home claims to a new triage model for six months, the remaining 90 percent keeping the old one. The challenger cuts average settlement time by 30 percent, but the files it routes to automatic handling show a 6 percent higher average cost: the team adjusts its threshold before rolling it out.
Follow the research
One letter a week. First, the week's underwriting note, when one is published: a dated, argued decision, signed by Paul Bernardski. Then what the observatory published on cyber risk, political risk and insurance. No account, no tracking.
No spam, one-click unsubscribe, no data ever sold.