The Transformer and Self-Attention

How Replacing Sequential Memory with All-to-All Queries Unlocked Modern Scale

Until June 2017, a machine reading a paragraph of text had to process it the way a clerk reads a telegraph ticker tape. Words entered the network one after another in strict chronological order. Recurrent neural networks (RNNs) and Long Short-Term Memory (LSTM) architectures passed a single fixed-width hidden state vector from word one to word two, then from word two to word three, and so on down the sentence.

This sequential design created two severe bottlenecks that throttled the scale of artificial intelligence. First, compressing a five-hundred-word page into a single narrow vector caused early details to fade out before the end of the passage. By the time the network reached the final sentence, the opening subject was blurred or forgotten. Second, sequential dependency starved modern parallel hardware. Because word fifty could not be computed until word forty-nine was finished, ten thousand graphics processing units (GPUs) could not split up a document and train on all its words simultaneously.

In June 2017, eight researchers at Google Brain and Google Research—Ashish Vaswani, Noam Shazeer, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin—published a conference paper titled Attention Is All You Need. Rather than trying to patch sequential memory, they discarded it completely. Their opening abstract proposed a simple network architecture called the Transformer built solely around attention mechanisms, dispensing with recurrent loops and convolutions altogether.

Instead of feeding words into the network one by one, the Transformer ingests an entire sequence of tokens in parallel. To preserve word order without recurrence, it stamps each word vector with a mathematical wave pattern—a positional encoding—that marks its exact coordinate in the sentence. At every layer, the network projects each word's vector into three distinct roles using learned weight matrices. It creates a Query vector ($Q$) representing what the word is looking for, a Key vector ($K$) advertising what information the word contains, and a Value vector ($V$) holding the content payload to be passed upward.

To decide how much every word in a passage should attend to every other word, the Transformer multiplies the matrix of Queries against the transposed matrix of Keys. It scales those inner products by the square root of the key dimension $d_k$, applies a softmax probability normalization, and multiplies the resulting weights by the Values:

Dividing by $\sqrt{d_k}$ is a vital stabilizing mechanic. In high dimensions, dot products between random vectors grow large in magnitude, pushing the exponential softmax function into flat saturation regions where calculus gradients vanish to near zero. Scaling by $\sqrt{d_k}$ keeps the variance near one so error signals flow cleanly during backpropagation. Furthermore, by splitting Queries, Keys, and Values across multiple parallel attention heads, a single layer tracks several relationships at once. One head links pronouns to nouns, a second tracks verb tense, and a third connects modifiers to objects.

Because matrix multiplications across all token positions happen simultaneously, Vaswani and his colleagues trained their English-to-German translation model to state-of-the-art accuracy in just three and a half days on eight Nvidia P100 GPUs. That run used a fraction of the compute time required by recurrent models. Because the path length between any two words in a Transformer is a single direct hop rather than a chain of sequential steps, a pronoun at the end of a ten-thousand-word contract can attend directly to the noun on page one.