Attention — kino

Attention and transformersabout 10 minutes

A Python dictionary does a hard lookup: give it a key, get exactly one value back. Attention is a soft lookup. You give it a query; it compares the query with every key, turns the similarities into weights that sum to 1, and returns the weighted average of the values. Because every step is differentiable, a network can learn what to look for.

scores_{j} = q \cdot k_{j}, α = softmax (scores), output = j \sum α_{j} v_{j}

Four keys, each with a value (a colour). Turn the query: weights shift toward the keys it points along, and the output blends their values.

Every ingredient is familiar: the dot product measures agreement (module 1), softmax turns scores into a distribution (module 2), and the output is a weighted average. In the translation setting of the last lesson, the query is the decoder's current state, and keys and values are the encoder's states for each source word.

Scaling by √d

With vectors of dimension $d$ whose entries are roughly independent with variance 1, a dot product is a sum of $d$ terms, so its variance is about $d$ . For $d = 512$ , scores spread over a range of tens, and softmax of large numbers is nearly one-hot: one weight near 1, the rest near 0, with gradients near zero. Dividing by $d$ brings the variance back to about 1. This is scaled dot-product attention, written for many queries at once as matrices:

Attention (Q, K, V) = softmax (\frac{Q K ^{⊤}}{d _{k}}) V

Shapes: $Q$ is $(n_{q}, d_{k})$ , $K$ is $(n_{k}, d_{k})$ , $V$ is $(n_{k}, d_{v})$ . Then $Q K^{⊤}$ is an $(n_{q}, n_{k})$ table of scores, the softmax runs along each row, and the result is $(n_{q}, d_{v})$ : one output per query. That table of weights is what people visualise as "attention maps".

try itedit the code and run it, nothing can break

python sleeps until you press run

Attention scores keys against a query with dot products, softmaxes the scores, and averages the values. Dividing by $d$ keeps the softmax from saturating.

← An RNN forward pass