Reinforcement learning from human feedback
A method tuning a model on compared human preferences rather than on correct answers, which relocates subjectivity onto the choice of annotators.
Ouyang and coauthors at OpenAI published InstructGPT in March 2022: raters preferred the answers of a 1.3-billion-parameter model fine-tuned on human feedback to those of the 175-billion-parameter GPT-3. A model trained to predict the continuation of text reproduces what is said, with no regard for what is useful or acceptable. Alignment tries to close that gap. For most requests there is no correct answer to show the model. The most widespread method therefore presents it with pairs of responses that human annotators have ranked. Those rankings train a reward model, which in turn is used to tune the main model. The chain is effective, and it relocates subjectivity without removing it. A few thousand annotators do the judging, recruited on criteria and guided by instructions that are almost never published; their judgments end up embedded in a system used by hundreds of millions of people. Most of the criticism bears on that opacity, more than on the technique. It matters directly to underwriting: behavior acquired through preferences cannot be documented like a rule, and cannot be audited by reading code.
The paper Training language models to follow instructions with human feedback, published in March 2022, described how this method was applied. It showed that human evaluators preferred a markedly smaller model tuned this way over a far larger model that was not.
Follow the research
One letter a week. First, the week's underwriting note, when one is published: a dated, argued decision, signed by Paul Bernardski. Then what the observatory published on cyber risk, political risk and insurance. No account, no tracking.
No spam, one-click unsubscribe, no data ever sold.