Shannon Information Entropy

How Bell Labs Measured Uncertainty Without Mentioning Heat

Inside the brick laboratories of Bell Telephone Laboratories in Murray Hill, New Jersey, in 1948, mathematician and electrical engineer Claude Shannon confronted a fundamental engineering problem. Telephone and telegraph engineers needed an exact mathematical yardstick to measure the quantity of information carried by any signal. Whether transmitting Morse code dots, human voice waves over copper lines, or television scan frames, they needed to know how much a signal could be compressed before losing content.

Shannon stripped away the subjective human meaning of words. Instead, he treated any message source as a discrete stochastic process that chooses symbols $x_1, x_2, \dots, x_n$ from a finite alphabet according to probabilities $P(x_1), P(x_2), \dots, P(x_n)$. If a symbol has probability 1.0, receiving it delivers zero surprise and zero new information. If a rare symbol has a tiny probability $P(x_i)$, its arrival resolves a large uncertainty. For two independent events to contribute additive information, the surprise of a single outcome must scale with $-\log_2 P(x_i)$.

In his July and October 1948 monograph in the Bell System Technical Journal, Shannon proved that only one mathematical function satisfies three basic axioms. It must be continuous in the probabilities, increase monotonically with the number of equally likely choices, and decompose consistently when choices are broken into sub-steps. That unique measure of average uncertainty per symbol is the Shannon entropy $H(X)$, which sets the minimum average codeword length $\bar{L}$ achievable by any lossless prefix code:

Measured in base-two logarithms, $H(X)$ gives the exact number of binary digits—or bits, a term suggested by Shannon's Bell Labs colleague John Tukey—required on average to represent each symbol emitted by the source. A fair coin toss with $P(\text{heads}) = P(\text{tails}) = 0.5$ has an entropy of $-\left(0.5\log_2 0.5 + 0.5\log_2 0.5\right) = 1.0$ bit. A heavily weighted coin that lands heads 99 percent of the time carries an entropy of just 0.0808 bits per flip. By encoding runs of multiple tosses together using Huffman or arithmetic coding, a thousand flips of that biased coin can be transmitted across a wire in roughly eighty-one binary pulses instead of a thousand.

Look closely at Shannon's sum and Boltzmann's gas formula. When all $W$ microstates of an isolated physical system have equal probability $P_i = 1/W$, Shannon's formula reduces to $\log_2 W$. It differs from Boltzmann's $S = k_B \ln W$ only by the base of the logarithm and the unit conversion constant $k_B \ln 2$. For nearly a decade, physicists debated whether this algebraic identity was a deep law of nature or a superficial pun.

In two landmark 1957 papers in the Physical Review, physicist Edwin T. Jaynes at Stanford University proved that the two entropies are the exact same concept. Jaynes showed that statistical mechanics does not depend on mechanical hypotheses about ergodic molecular trajectories. Instead, the canonical Boltzmann distribution $P_i \propto e^{-E_i / k_B T}$ follows directly from maximizing Shannon information entropy subject to the macroscopic constraint of known average energy $\langle E \rangle$. Thermodynamic entropy is simply the Shannon missing information required to specify the exact microstate of a physical system given its macroscopic gauge readings. In 1951, Claude Shannon asked volunteers at Bell Laboratories in Murray Hill to guess twenty-seven-character passages of ordinary English prose letter by letter. Subjects guessed the exact next character on their very first try sixty-nine percent of the time, demonstrating that statistical correlations strip away roughly three-quarters of the raw alphabetical entropy.