Next-token prediction

A language model assigns a probability to any sequence of tokens. The chain rule of probability lets us write that as a product of next-token predictions, each conditioned on everything before it:

So a model only ever needs to answer one question: given this context, what is the probability distribution of the next token? Generate by sampling a token, appending it, and asking again. That's how every chat model produces text, one token at a time.

n-gram models

The oldest approach, going back to Claude Shannon's 1948 experiments with English text, conditions on only the last $n-1$ tokens and estimates probabilities by counting. A bigram model uses one token of context: $P(\text{next}\mid \text{current})=\text{count(current, next)}/\text{count(current)}$.

• Counts of zero give probability zero, and an infinite loss on any test text that contains the pair. Smoothing (add a small count to every cell) fixes this; it's regularisation again.
• Longer contexts are more predictive but the table grows exponentially: with a 50,000-token vocabulary, a 4-gram table has $50,000^4\approx 6\times 10^{18}$ cells, almost all never observed.
• Neural language models (Bengio et al., 2003) replace the table with a network that generalises across similar contexts via embeddings. Transformers extend the context to thousands or millions of tokens.

The loss, and perplexity

Training minimises the average negative log-probability of the actual next token, which is exactly the cross-entropy loss from module 2, applied at every position. Every sentence in the training data provides as many training examples as it has tokens, with labels for free. That is why language models can learn from raw text at enormous scale: self-supervised learning.

Perplexity is the exponentiated loss, and it has an intuitive reading: a perplexity of 20 means the model is, on average, as uncertain as if it were choosing uniformly among 20 tokens. A model that knows nothing about a 100-token vocabulary has perplexity 100.

Predicting the next token sounds modest, but doing it well across all human text requires modelling grammar, facts, arithmetic, the intentions of characters in a story, and the style of a legal contract. That's the bet behind large language models: a simple objective, pushed far enough, forces the model to learn a great deal about the world that produced the text.

A language model predicts the next token's probability distribution. Its loss is cross-entropy on the true next token; perplexity is that loss exponentiated.