Context window
The maximum amount of text a model can consider at once, an engineering constraint that directly drives the cost and confidentiality of a use case.
Liu and coauthors published "Lost in the Middle" in 2023: the models they tested retrieve information well at the start or end of the context and much less well in the middle. The context window is the maximum amount of text a model processes at once, taken in and considered before any reasoning. Within a few years it grew from a few thousand units to several hundred thousand, and the way applications are built followed. Developers used to split documents and present only extracts, which required a prior search and inherited its errors. Now they present the whole document. Three practical consequences follow, and they bear directly on underwriting a risk. The cost of a call grows with the amount of context supplied, so a large window invites expensive uses no budget anticipated. Confidentiality changes scale, since an entire contract goes out instead of an extract. And reliability does not follow linearly: evaluations show that attention to information placed in the middle of a very long context declines, so a large window is no guarantee that everything was read.
Google's announcement in February 2024 of a one million token context window for its Gemini 1.5 model marked a jump of roughly two orders of magnitude over the models in wide use two years earlier.
Follow the research
One letter a week. First, the week's underwriting note, when one is published: a dated, argued decision, signed by Paul Bernardski. Then what the observatory published on cyber risk, political risk and insurance. No account, no tracking.
No spam, one-click unsubscribe, no data ever sold.