RNNs, LSTMs & Word Embeddings

Sequences need memory

Text, speech, sensor streams, DNA: order matters. "Dog bites man" ≠ "man bites dog". A feed-forward net sees a fixed-size bag of numbers with no notion of order or variable length. Sequence models fix this.

First: how do words become numbers?

One-hot vectors (ML path) treat "cat" and "kitten" as unrelated as "cat" and "carburetor". Word embeddings instead assign each word a dense learned vector (~300 dims) such that words in similar contexts get similar vectors — learned (word2vec, 2013) by training a tiny network to predict neighboring words on billions of sentences. The famous result: embedding geometry captures meaning, even analogies:

$\vec{\text{king}} - \vec{\text{man}} + \vec{\text{woman}} \approx \vec{\text{queen}}$

Meaning as geometry — the embedding worldview from the ML path, now learned. Every modern language model begins with an embedding layer.

Recurrent networks: a loop with a memory

An RNN reads a sequence one token at a time, maintaining a hidden state $\mathbf{h}$ — a running summary of everything read so far:

$\mathbf{h}_t = g\!\left(W_h \mathbf{h}_{t-1} + W_x \mathbf{x}_t + \mathbf{b}\right)$

Same weights every step (weight sharing across time, like CNNs share across space), any sequence length, output whenever needed: a sentiment label at the end, a translation word by word, or the next token in a text — the training objective that will grow into LLMs.

The flaw: training unrolls the loop into a very deep chain — one layer per timestep — and backpropagation multiplies derivatives across it. A 100-word sentence = a 100-layer network: vanishing gradients return with a vengeance. Plain RNNs forget what they read a few dozen steps ago; the signal linking "the cats … were" across a long clause simply dies.

LSTM: a memory with gates

The Long Short-Term Memory cell (1997!) engineers a fix: a separate cell state — a conveyor belt running through time nearly untouched — managed by three learned gates (sigmoids in [0,1] acting as differentiable valves): the forget gate decides what to erase from the belt, the input gate what to write, the output gate what to reveal. Gradients ride the belt with barely any decay — skip connections through time, morally the same medicine as ResNet. LSTMs powered the 2014–2017 golden age: Google Translate, Siri's speech recognition, early text generation.

Remaining bottlenecks: reading is strictly sequential (can't parallelize on GPUs — slow to train at scale), and an entire book must squeeze through one fixed-size state vector. A model that could look back at any earlier word directly would fix both. That idea — attention — is the next lesson, and it ended the RNN era.

Key takeaways

• Embeddings: words as learned vectors where geometry ≈ meaning.
• RNNs carry a hidden state through time; depth-in-time revives vanishing gradients; LSTM gates fix it.
• Sequential reading and the fixed-size bottleneck remained — setting the stage for attention.