The Measure of Surprise

Why Information Has Nothing to Do With Meaning and Everything to Do With Choice

Before 1948, telephone, radio, and television engineers believed that every communication medium obeyed its own separate physical rules. Telegraph operators counted words per minute in Morse dots and dashes. Telephone engineers measured analog voice bandwidth in kilohertz, while television designers tracked scanning lines and video carrier waves. Nobody knew how to compare the information content of a symphony broadcast over FM radio against a stack of teletype telegrams or a black-and-white photograph. Worse, natural philosophers kept tangling the engineering problem of signal transmission with the psychological question of what a message meant to a human listener.

Working largely alone in his closed-door office at Bell Telephone Laboratories across the 1940s, Claude Shannon unified the entire field in his two-part paper A Mathematical Theory of Communication. Published in the July and October 1948 issues of the Bell System Technical Journal, the paper severed information from human semantics in its opening paragraphs. A telegraph wire does not care whether a twenty-word telegram announces a historic peace treaty or repeats a trivial grocery list. What matters to the engineering system is that the transmitted message was chosen from a well-defined set of possible messages.

Building on earlier logarithmic formulas proposed at Bell Labs by Harry Nyquist in 1924 and Ralph Hartley in 1928, Shannon modeled any information source as a stochastic process. At each step, the source selects symbol $i$ from an alphabet of $n$ choices with probability $p_i$. Adopting the word bit—coined during a Bell Labs lunch conversation by statistician John Wilder Tukey as a contraction of binary digit—Shannon asked how to measure the average uncertainty or surprise produced by each symbol choice.

He wrote down three simple qualitative requirements that any sensible measure of uncertainty $H(p_1, p_2, \dots, p_n)$ must satisfy. First, $H$ must be a continuous function of the probabilities $p_i$. Second, if all $n$ symbols are equally likely ($p_i = 1/n$), then $H$ must increase monotonically as the number of choices $n$ grows. Third, if a single multi-way choice is broken into a tree of two successive sub-choices, the total uncertainty must equal the weighted sum of the individual uncertainties. Shannon then proved mathematically that up to a positive scaling constant fixing the unit of measurement, only one function in all of mathematics satisfies those three axioms:

Because that expression takes the exact functional form of Ludwig Boltzmann's and J. Willard Gibbs's thermodynamic entropy $S = -k_B \sum p_i \ln p_i$ in statistical mechanics, Shannon named the quantity entropy. Suppose a source is completely predictable, such as a rigged coin that lands heads with probability $p_1 = 1$ and tails with $p_2 = 0$. Then $H = 0$ bits, because receiving the message resolves zero uncertainty. Uncertainty peaks at one bit per toss only when the coin is fair ($p_1 = p_2 = 0.5$).

To illustrate how statistical structure lowers entropy without invoking human grammar, Shannon constructed a sequence of stochastic Markov approximations to written English using twenty-seven symbols. Picking letters at random with equal probability yielded unpronounceable gibberish (XFOML RXKHRJFFJUJ). Next, he sampled letters according to their actual English frequencies (E at twelve percent, T at nine percent), and then according to two-letter digram transitions. Finally, sampling from three-letter trigram and two-word transition tables produced pronounceable strings that resembled real English syntax purely from probability weights. By stripping semantic meaning out of engineering calculations, Shannon gave every medium on Earth—from copper telegraph wires and vinyl phonograph grooves to cellular DNA and optical fibers—a single universal unit of measurement in the binary digit.