Training a smaller model to reproduce a larger one's outputs, which cuts running cost and shifts the question of ownership.
Distillation trains a small model, called the student, to reproduce not the correct answers but the full outputs of a larger model, called the teacher, including the probability distribution it assigns to the answers it rejected. That extra information is what makes the technique effective, the student learning from the teacher's implicit reasoning rather than only from its verdict. The economic value is considerable, since a model's running cost is paid at every call while its training is paid once, so a student ten times smaller and nearly as good changes the economics of an entire service. The technique raises in return an ownership question the law has not settled: a model distilled from the outputs of a paid service reproduces a capability without copying any parameter, so there is neither software infringement nor database extraction in the classical sense. Providers have therefore handled the matter contractually, forbidding in their terms of use the employment of outputs to train a competing model.
The method was formalized in the paper Distilling the Knowledge in a Neural Network, published in 2015 by Geoffrey Hinton, Oriol Vinyals and Jeff Dean, which showed that a reduced model trained on the full outputs of an ensemble retained most of its performance.
knowledge distillation, distillation des connaissances, modèle élève, student model