Attention & Transformers

Attention: just look it up

In "The animal didn't cross the street because it was too tired", what does "it" refer to? You resolved that by glancing back at "animal" — not by replaying the sentence through a memory bottleneck. Attention gives networks that ability: every word may look directly at every other word and pull in what's relevant.

The mechanism is a soft database lookup. Each word's embedding is projected (by learned matrices) into three vectors:

• Query $\mathbf{q}$ — "what am I looking for?"
• Key $\mathbf{k}$ — "what do I contain?"
• Value $\mathbf{v}$ — "what do I contribute if selected?"

Word A attends to word B in proportion to how well A's query matches B's key — a dot product (the similarity measure from the math path). Softmax turns the scores into weights, and each word's new representation is the weighted average of all values:

$\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^T}{\sqrt{d_k}}\right) V$

That's the complete famous formula: similarity scores → scale → probabilities → weighted lookup. "It" learns a query matching animal-like keys, and its representation absorbs "animal". Multi-head attention runs several of these lookups in parallel — one head might track grammar, another coreference, another nearby adjectives — and concatenates the results.

The transformer

The 2017 paper "Attention Is All You Need" removed recurrence entirely. A transformer block is:

x → LayerNorm → Multi-Head Attention → (+x)
  → LayerNorm → small MLP           → (+x)

…with skip connections (ResNet's gift) around each part; stack the block dozens of times. Since attention itself ignores word order, positional encodings are added to the embeddings so the model knows token positions.

Why it conquered everything:

1. Parallelism — no sequential loop: all tokens process simultaneously as giant matrix multiplications. GPUs feast; training scales to internet-sized corpora.
2. Direct long-range access — token 1 reaches token 10,000 in one hop; no memory bottleneck, no vanishing path.
3. Generality — treat anything as a token sequence: image patches (Vision Transformers), audio, protein residues (AlphaFold), game moves. One architecture, every domain.

Scaling into GPT

Train a transformer on one objective — predict the next token — over trillions of words, and something remarkable happens: grammar, facts, translation, reasoning patterns emerge, because predicting text well requires modeling the world that produced it. GPT-style models are "just" enormous transformers (hundreds of layers/billions of parameters) trained this way, then aligned to be helpful assistants. The scaling laws — performance rising predictably with model size, data and compute — turned AI progress into an engineering roadmap. The full story (pretraining, RLHF, prompting, limitations) awaits in the AI path.

Key takeaways

• Attention = soft lookup: query·key similarities → softmax → weighted sum of values.
• Transformer = attention + MLP + LayerNorm + skip connections, stacked; positional encodings supply order.
• Parallelism + long-range access + token universality → the architecture of modern AI, GPT included.