A large language model starts as a decoder-only transformer (module 5) trained on next-token prediction over an enormous corpus: filtered web crawl, books, code, scientific papers, encyclopaedias. This phase, pretraining, produces a base model: a remarkably capable text continuer that has no particular wish to be helpful.
- Data is curated heavily: deduplication, quality filters, removal of personal information and toxic content, and careful mixing of sources. Data quality is one of the biggest levers on the final model.
- Scale: modern pretraining runs process trillions of tokens on thousands of GPUs for weeks to months.
- Compute for a transformer is about floating-point operations for parameters and training tokens: about 2 per parameter per token for the forward pass, 4 for the backward.
Scaling laws
The striking empirical finding of Kaplan et al. (2020) is that test loss falls as a smooth power law in model size, dataset size and compute, over many orders of magnitude, provided none of them is the bottleneck. A power law is a straight line on a log-log plot: every 10× more compute buys a predictable drop in loss. This let labs forecast the result of a huge training run from small ones.
Hoffmann et al. (2022), the Chinchilla paper, refined the question: for a fixed compute budget, how should you split it between parameters and data? They fitted a loss of the form below and found that earlier models were too big for the amount of data they saw. Their rule of thumb: compute-optimal training uses roughly 20 tokens per parameter. Their 70B-parameter Chinchilla, trained on 1.4 trillion tokens, outperformed the 280B-parameter Gopher trained with the same compute.
Since then, many models have been trained far beyond 20 tokens per parameter on purpose: a smaller model trained longer costs more to train but much less to run, and serving cost dominates once millions of people use it.
In-context learning
GPT-3 (Brown et al., 2020), with 175 billion parameters, showed something new at scale: give the model a few examples of a task in the prompt ("English: cheese → French: fromage") and it continues the pattern for new inputs, with no weight updates at all. This few-shot, in-context learning made a single pretrained model usable for thousands of tasks through prompting alone.
Pretraining is next-token prediction at enormous scale. Loss falls as a power law in parameters, data and compute; balancing parameters and data (about 20 tokens per parameter) is compute-optimal.