The presence of evaluation material in the training data, so that memorization is measured where capability is assumed.
A model trained on a large fraction of the web has very probably encountered the public question sets later used to score it, since those sets are published online with their answer keys. The resulting score then mixes two things of opposite nature, generalization and memorization, and nothing in the number separates them. The practical consequence is that a few points between two models on a public benchmark says almost nothing, while the same test rebuilt with unseen questions often produces a different ranking. Remedies exist and are all imperfect: canary strings inserted into test sets to detect their presence in a corpus, continuously refreshed benchmarks, or private evaluations whose questions are never published, the last protecting the measurement at the cost of reproducibility. For an insurer assessing the reliability of a system, the lesson is never to accept a public score as evidence of performance in a given domain, and to require evaluation on the insured's own real data.
Work published from 2023 onward on contamination of large language model benchmarks showed that several widely cited reference tests appeared, in whole or in part, in the public training corpora used at that time.
benchmark contamination, fuite de jeu de test, data contamination, contamination des tests