Transformer architecture
A neural network architecture based on the attention mechanism, published in 2017, underlying large language models, whose ability to process long contexts transformed document analysis.
The transformer architecture was presented in 2017 by a research team in a paper titled Attention Is All You Need. It relies on the attention mechanism, which lets each element of a sequence, for example each word of a text, weigh the importance of all the others to build its representation. Unlike the recurrent architectures before it, which read text word by word, it processes the whole sequence in parallel, making it possible to train much larger models on considerable volumes of data. It underlies large language models, but also models of images, sound and time series. For insurance, its importance lies in two properties. Its ability to connect distant elements in a text makes it possible to analyze long policies and reports, where an exclusion on page thirty modifies a cover on page three. And its cost grows quickly with context length, which explains context window limits and the sector's dependence on a few providers able to fund training, a source of concentration and single-provider risk.
Textbook case: an assistant built on a transformer model, an architecture published in June 2017, analyzes a 60-page policy and notes that a cyber event definition in an appendix narrows the scope of the business interruption cover described in the first part, a link a keyword search tool reading passages separately had missed.
Follow the research
One letter a week. First, the week's underwriting note, when one is published: a dated, argued decision, signed by Paul Bernardski. Then what the observatory published on cyber risk, political risk and insurance. No account, no tracking.
No spam, one-click unsubscribe, no data ever sold.