Model distillation
Training a smaller model to reproduce a larger one's outputs, which cuts running cost and shifts the question of ownership.
March 2015: Geoffrey Hinton, Oriol Vinyals and Jeff Dean, at Google, published "Distilling the Knowledge in a Neural Network". They called "soft targets" the probabilities a large model also assigns to wrong answers, and used them to build a single acoustic model for a heavily used commercial system. Distillation trains a small model, the student, on the full outputs of a larger model, the teacher, and not only on the correct answers: the student also sees the probability the teacher gives to the answers it rejects. That extra information is what makes the technique work. The student learns the teacher's implicit reasoning as well as its verdict. The economic value is considerable. Training is paid for once, while running a model is paid for at every call; a student ten times smaller and nearly as good therefore changes the economics of an entire service. In return, the technique raises an ownership question the law has not settled. A model distilled from the outputs of a paid service reproduces a capability without copying a single parameter, so there is neither software infringement nor database extraction in the classical sense. Providers have therefore handled the matter by contract, and their terms of use forbid using outputs to train a competing model.
The method was formalized in the paper Distilling the Knowledge in a Neural Network, published in 2015 by Geoffrey Hinton, Oriol Vinyals and Jeff Dean, which showed that a reduced model trained on the full outputs of an ensemble retained most of its performance.
Follow the research
One letter a week. First, the week's underwriting note, when one is published: a dated, argued decision, signed by Paul Bernardski. Then what the observatory published on cyber risk, political risk and insurance. No account, no tracking.
No spam, one-click unsubscribe, no data ever sold.