A tiny language model

This is a large language model in miniature. Same objective (predict the next token), same loss (cross-entropy), same sampling (temperature), same evaluation (held-out perplexity), but the tokens are letters, the corpus is 778 first names, and the model fits on a screen.

The approach follows Andrej Karpathy's makemore lectures: start with counting, then show that a one-layer neural network trained by gradient descent learns the same thing, then add context.