Carbon footprint of model training
The emissions attached to a model's training compute, whose magnitude depends more on the local electricity mix than on the size of the model.
November 2022: Sasha Luccioni, Sylvain Viguier and Anne-Laure Ligozat put the final training run of BLOOM, a 176-billion-parameter model, at 24.7 metric tons of CO2 equivalent. It ran on Jean Zay, on a French grid emitting 57 grams per kilowatt-hour, against 231 for the grid that trained OPT. Training a large model uses electricity, and the electricity mix of the place and the hour decides what emissions come out of it. Two identical training runs can therefore differ by a substantial factor depending on the region chosen. That shifts the question: an operator cuts its footprint more by choosing where to compute than by computing less. A second distinction matters and is often left out. Training is paid for once, inference at every use; as soon as a model is widely deployed, cumulative inference exceeds training, and attention moves from announcements to daily operation. For insurance, this is less a peril than a transition and reputational risk. A company that publishes a reduction pathway while deploying compute-hungry services must account for the gap, and its indirect emissions include those of the compute provider it uses.
The paper Carbon Emissions and Large Neural Network Training was published in 2021 by researchers at Google and Berkeley. It put the emissions of training GPT-3 at around 550 metric tons of carbon dioxide equivalent. It also showed that the choice of location and electricity mix changed that figure by an order of magnitude.
Follow the research
One letter a week. First, the week's underwriting note, when one is published: a dated, argued decision, signed by Paul Bernardski. Then what the observatory published on cyber risk, political risk and insurance. No account, no tracking.
No spam, one-click unsubscribe, no data ever sold.