Reinforcement learning from human feedback

A model trained to predict the continuation of text reproduces what is said rather than what is useful or acceptable, a gap alignment seeks to close. The most widespread method does not present the model with correct answers, which do not exist for most requests, but with pairs of responses ranked by human annotators. Those rankings train a reward model, which in turn is used to tune the main model. The chain is effective and it relocates subjectivity without removing it: the judgments of a few thousand annotators, recruited on criteria and guided by instructions that are almost never published, end up embedded in a system used by hundreds of millions of people. That opacity is what most criticism bears on, more than the technique itself. It matters directly to underwriting, since behavior acquired through preferences cannot be documented like a rule and cannot be audited by reading code.

The paper Training language models to follow instructions with human feedback, published in March 2022, described the application of this method and showed that a markedly smaller model tuned this way was preferred by human evaluators over a far larger model that was not.

RLHF, reinforcement learning from human feedback, alignement par préférences