Kullback-Leibler Divergence and Cross-Entropy

How Cryptanalysis Built the Loss Function That Trains Modern Neural Networks

During World War II at the U.S. Army Signal Intelligence Service headquarters at Arlington Hall, Virginia, mathematician Solomon Kullback led the statistical research section breaking Japanese military ciphers alongside fellow cryptanalyst Richard Leibler. Every day, intercept analysts faced a concrete decision problem. Given a short stream of intercepted symbols $x_1, x_2, \dots, x_k$, how much statistical weight does each observation provide to discriminate between Hypothesis $P$ and Hypothesis $Q$? Hypothesis $P$ states that the message came from a structured cipher key, while Hypothesis $Q$ states that the stream is random background noise.

Returning to civilian research at George Washington University and the National Security Agency after the war, Kullback and Leibler generalized their wartime log-likelihood ratios in a March 1951 paper in the Annals of Mathematical Statistics. Suppose a physical environment or language source actually generates events according to a true probability distribution $P(x)$. Meanwhile, an observer, receiver, or neural model encodes those events using an approximate internal probability distribution $Q(x)$. By Shannon's source coding theorem, an optimal code designed for the true distribution $P$ requires an average of $H(P) = -\sum P(x)\log P(x)$ bits per symbol. If we instead build our code using the flawed beliefs $Q(x)$, each symbol $x$ is assigned a codeword of length $-\log Q(x)$. That mismatch costs a larger average length known as the cross-entropy $H(P, Q)$. The exact penalty of wasted bits caused by the mismatch between reality $P$ and model $Q$ is the Kullback-Leibler divergence $D_{\text{KL}}(P \parallel Q)$:

By Gibbs's inequality, $D_{\text{KL}}(P \parallel Q)$ is strictly non-negative. It equals zero if and only if the model matches reality everywhere ($Q(x) = P(x)$). Crucially, Kullback and Leibler noted that $D_{\text{KL}}$ is not a symmetric geometric distance—$D_{\text{KL}}(P \parallel Q) \neq D_{\text{KL}}(Q \parallel P)$—because the cost of being surprised depends on which distribution actually governs the world. If an event $x$ occurs in reality ($P(x) > 0$) and your model claims it is impossible ($Q(x) = 0$), the divergence blows up to infinity.

Look at the rearrangement $H(P, Q) = H(P) + D_{\text{KL}}(P \parallel Q)$. That simple additive identity is the mathematical engine behind modern artificial intelligence. When engineers train a deep neural network or a large language model to classify images or predict the next word in a sentence, the true entropy $H(P)$ of the training dataset is a fixed constant that does not depend on the network's synaptic weights $\theta$. Therefore, adjusting the weights via stochastic gradient descent to minimize the cross-entropy loss $H(P, Q_\theta)$ is mathematically identical to driving the Kullback-Leibler divergence $D_{\text{KL}}(P \parallel Q_\theta)$ down toward zero. Minimizing cross-entropy forces the network's internal probability distribution $Q_\theta$ to converge onto the statistical structure of reality $P$.

In 1994, computer scientists Geoffrey Hinton and Richard Zemel at the University of Toronto proved that this information penalty obeys the exact same algebraic equation as Hermann von Helmholtz's 1882 thermodynamic free energy $F = E - TS$. In variational Bayesian inference and Karl Friston's active-inference brain models, an exact posterior distribution $P(z \mid x)$ over hidden environmental causes $z$ is often computationally intractable. Instead, the system adjusts an internal recognition density $Q(z)$ to minimize variational free energy, simultaneously maximizing evidence and shrinking $D_{\text{KL}}(Q \parallel P)$. In 2020, OpenAI trained the 175-billion-parameter GPT-3 language model across three hundred billion tokens of text using cross-entropy loss over a vocabulary of 50,257 byte-pair tokens. Driving validation cross-entropy down from an initial uniform-guess penalty of 10.82 nats per token to 2.05 nats per token transformed the network from outputting gibberish into writing coherent prose.