Model evaluations (evals)
Sets of representative cases, with expected answers and scoring criteria, that measure a language model's quality on a specific task before and after each change, the only serious proof that an assistant works.
A language model evaluation is a set of cases representative of a real task, with the expected answer or criteria for judging an answer, on which the system is run to measure its performance. Public benchmarks measure general capabilities; they say little about an assistant's quality on a specific insurance task, especially as their questions have sometimes been seen during training. An organization deploying a model must therefore build its own evaluations: extracting the limits and deductibles from a hundred real policies, classifying claims, answering questions on underwriting policy, with hard cases and traps. Scoring can be exact, by comparison with an expected value, human, by experts, or delegated to another model asked to judge against a rubric, a fast method that must itself be checked. Evaluations serve to choose a model, measure the effect of an instruction or version change, and detect regressions. For the insurer, they are the performance documentation supervisors expect, and the condition for a deployment to rest on measurement rather than impression.
Textbook case: before replacing the model version behind its underwriting assistant, an insurer reruns its 300 evaluation cases. The new version gains 6 points on limit extraction, but invents a nonexistent deductible in 4 percent of policies without one, against 1 percent before: deployment is suspended until fixed.
Follow the research
One letter a week. First, the week's underwriting note, when one is published: a dated, argued decision, signed by Paul Bernardski. Then what the observatory published on cyber risk, political risk and insurance. No account, no tracking.
No spam, one-click unsubscribe, no data ever sold.