The Problem Attention Solved

Why RNNs were the bottleneck and how attention broke it open

Learning objectives

Understand gradient vanishing in RNNs and why long-range dependencies are hard
See why sequential processing prevents parallelisation
Grasp the core O(1) path-length intuition of attention

Visual Explainer

RNN — Sequential (slow)

Transformer — Parallel (fast)

Left: token 1 must reach token 5 through 4 hops. Right: direct connection, one step.

Before we write a single line of Transformer code, we need to understand why it was invented. The 2017 paper didn't appear from nowhere — it solved two concrete, painful problems that researchers had been fighting for years.

The State of the Art in 2017

By 2017, the best machine translation models used the encoder-decoder RNN with attention (Bahdanau et al., 2015). The encoder read the input sentence token-by-token, accumulating information into a fixed-size hidden state. The decoder then generated the output one token at a time, using the encoder's hidden state as context.

This worked — but it had two fatal flaws that capped how far it could scale.

Problem 1: Vanishing Gradients and the Long-Range Memory Problem

An RNN processes a sequence step by step. At each step t, the hidden state is:

h t = tanh(W h \cdot h t-1 + W x \cdot x t)

To compute the gradient of the loss with respect to the hidden state at step 1, you must multiply through every step between 1 and T:

\partialL/\partialh 1 = \partialL/\partialh T \cdot \prod t=2..T \partialh t /\partialh t-1

Each factor in that product is a Jacobian matrix. If the weights are small (which they must be to prevent exploding activations), those Jacobians have spectral radius < 1. Multiply 50 of them together and the gradient becomes numerically zero. The model simply cannot learn relationships between tokens that are far apart.

In practice this means an RNN translating "The animal didn't cross the street because it was too tired" struggles to connect "it" back to "animal" if there are many tokens in between. Humans do this effortlessly. RNNs don't.

Problem 2: You Can't Parallelize a Chain

GPUs are good at one thing: running thousands of operations simultaneously. An RNN defeats this completely. Computing h_t requires h_t-1, which requires h_t-2, and so on. It's a serial dependency chain — step 2 literally cannot start until step 1 finishes.

On a sequence of 50 tokens, your GPU has ~5,000 cores sitting idle while each serial step runs. Training a large RNN on a billion tokens takes weeks. This wasn't just annoying — it meant larger models weren't worth building.

The Insight: Direct Connections Between All Positions

What if, instead of passing information through a chain of hidden states, every token could directly look at every other token in a single operation?

That's attention. Given a sequence of 5 tokens, we compute a 5×5 matrix of "how much should token i look at token j?" — all at once, in parallel. The path length between any two tokens drops from O(n) to O(1). And the whole computation is one matrix multiplication, which GPUs do brilliantly.

The Transformer paper's central claim was radical in 2017: you don't need recurrence or convolutions at all. Attention alone is sufficient — and faster, and better.

The key insight to carry forward: Attention replaces the O(n) information path of an RNN with O(1) direct connections between all pairs of positions. This makes long-range dependencies easy to learn and enables full parallelisation during training.

Interactive Code

python · numpy · matplotlib

import numpy as np
import time

# Demonstrate why RNNs can't parallelise
seq_len = 10
d_hidden = 64
d_input = 32
np.random.seed(42)

X = np.random.randn(seq_len, d_input)
W_h = np.random.randn(d_hidden, d_hidden) * 0.05
W_x = np.random.randn(d_input, d_hidden) * 0.1

# RNN: strictly sequential
h = np.zeros(d_hidden)
rnn_hiddens = []
for t in range(seq_len):
    h = np.tanh(h @ W_h + X[t] @ W_x)
    rnn_hiddens.append(h.copy())

print("RNN: processed", seq_len, "tokens, ONE at a time")
print("Info from position 0 must survive", seq_len-1, "matrix multiplications")

# Measure how much info from position 0 survives to position seq_len-1
W_h_power = np.linalg.matrix_power(W_h, seq_len - 1)
survival = np.linalg.norm(W_h_power)
print(f"Gradient norm after {seq_len-1} steps: {survival:.6f}")
print()

# Attention: all pairs in ONE operation
W_q = np.random.randn(d_input, 32) * 0.1
W_k = np.random.randn(d_input, 32) * 0.1

Q = X @ W_q  # (seq_len, 32)
K = X @ W_k  # (seq_len, 32)
scores = Q @ K.T  # (seq_len, seq_len) — ALL pairs at once

print("Attention: computed", seq_len, "×", seq_len, "=", seq_len**2, "pair scores in ONE matmul")
print("Scores shape:", scores.shape)
print("Every position can directly see every other position — path length = 1")

Exercises

Modify the code to show that attention's O(1) path length is independent of sequence length. Try seq_len = 5, 50, 500.

Key Takeaway

Attention gives every token a direct O(1) path to every other token, solving vanishing gradients and enabling full GPU parallelism.

Up next

Now we'll see how words become the vectors that attention operates on.