Benchmark contamination

May 2024: Scale AI released GSM1k, a set of new arithmetic problems modeled on the public GSM8k set. Several model families, Phi and Mistral first among them, lost accuracy markedly on it. The gap grew with the likelihood that a model would reproduce GSM8k examples verbatim. A model trained on a large share of the web has very probably already seen the public question sets later used to score it, since those sets are published online with their answer keys. Its score then mixes two things of opposite nature, generalization and memorization, and nothing in the number separates them. In practice, a few points between two models on a public benchmark say almost nothing. The same test rebuilt with unseen questions often produces a different ranking. Remedies exist, and all are imperfect: canary strings inserted into test sets to detect them in a corpus, continuously refreshed benchmarks, or private evaluations whose questions are never published. The last protects the measurement at the cost of reproducibility. An insurer assessing the reliability of a system should therefore never accept a public score as evidence of performance in a given domain. It should require an evaluation on the insured's own real data.