The transformer block

Attention and transformersabout 11 minutes

The transformer (Vaswani et al., 2017, "Attention Is All You Need") replaced recurrence with attention entirely. It was designed for translation and became the architecture behind essentially every large language model, and much of modern vision, speech and protein modelling too.

Position information

Self-attention is permutation-equivariant: shuffle the input tokens and the outputs shuffle the same way. On its own it can't tell "dog bites man" from "man bites dog". So position is added to each token's embedding. The original paper used fixed sinusoidal encodings, sines and cosines of different frequencies; many models learn position embeddings instead, and recent ones such as LLaMA rotate queries and keys by position (RoPE, Su et al., 2021).

P E_{(p os, 2 i)} = sin (\frac{p os}{1000 0 ^{2 i / d}}), P E_{(p os, 2 i + 1)} = cos (\frac{p os}{1000 0 ^{2 i / d}})

Sinusoidal position encodings: each row is a position, each column a dimension. Low dimensions oscillate fast, high ones slowly, like the hands of a clock.

The block

A transformer is a stack of identical blocks. In the pre-norm form used by GPT-2 and most later models, each block does two things, each wrapped in a residual connection:

x x \leftarrow x + MultiHeadAttention (LayerNorm (x)) \leftarrow x + MLP (LayerNorm (x))

One pre-norm transformer block. Information flows up the residual stream; each sub-layer reads from it and adds back a correction.

Attention moves information between positions.
The MLP (two dense layers, hidden size usually 4× the model width, with a GELU) processes each position independently. Most of the parameters live here.
Residual connections create a "residual stream" that every layer reads from and writes to, and keep gradients healthy through dozens of layers.
Layer normalisation standardises each token's vector, keeping activations in a stable range.

Three families

type	attention	examples	good at
encoder-only	bidirectional (sees everything)	BERT (2018)	understanding: classification, search embeddings
decoder-only	causal (sees the past)	GPT series, LLaMA, Claude	generation, and nearly everything via prompting
encoder-decoder	encoder bidirectional, decoder causal + cross-attention	original transformer, T5	translation, summarisation

A useful rule of thumb for decoder-only models: with $L$ layers of width $d$ , the parameter count is about $12 L d^{2}$ (attention has about $4 d^{2}$ per layer, the MLP about $8 d^{2}$ ), plus the embeddings. GPT-2 small ( $L = 12$ , $d = 768$ ) gets about 85 million from that formula, plus 38 million for token embeddings, for its 124 million total.

try itedit the code and run it, nothing can break

python sleeps until you press run

A transformer adds position to token embeddings, then stacks blocks of attention (between tokens) and MLPs (within tokens), each with a residual connection and layer norm.

← Self-attention and multi-head attention