Algorithmic black box

Cynthia Rudin, Nature Machine Intelligence, May 2019, "Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead": for high-impact decisions, a model that is readable from the start beats an explanation bolted onto an opaque one. The algorithmic black box is a system, typically a deep neural network, whose inputs and outputs can be observed but whose internal path to a given decision remains largely unexplainable, even to its designers. No trade secret causes this opacity. It comes from the very nature of these models, whose decisions emerge from the interaction of billions of parameters in very high-dimensional vector spaces, with no interpretable symbolic representation. For risk management and agency theory, this opacity creates a new kind of information asymmetry. The human principal, whether manager or insurer, keeps control of intent but can no longer audit the reasoning chain that produced a result. The AlgoPolis paper shows that this structural blindness replaces the classic moral hazard of the human employee with technological opacity. It also drives up supervision costs to the point of threatening the efficiency automation promises. Explainability research, known as XAI, tries to make decisions more interpretable, but its progress remains partial, and the black box remains one of the main obstacles to using AI in regulated fields and to insuring it.