A language model doesn't see letters or words; it sees tokens, integer ids from a fixed vocabulary. How text is cut into tokens is a design decision with surprising consequences, and nearly every modern model makes the same choice: subword tokens.
| unit | vocabulary | problem |
|---|---|---|
| characters | small (≈100–300) | sequences are long, and the model must learn spelling from scratch |
| words | huge (millions) | rare, misspelled and new words become an "unknown" token |
| subwords | medium (≈30k–200k) | the compromise: frequent words are one token, rare ones split into pieces |
Byte-pair encoding
Byte-pair encoding (BPE) started as a 1994 data-compression algorithm (Gage) and was adapted for neural translation by Sennrich, Haddow and Birch in 2016. Training it is a simple loop:
- Start with every word split into characters (or, in GPT-2 onward, raw bytes, so any text at all can be encoded).
- Count every adjacent pair of symbols across the training corpus.
- Merge the most frequent pair into a new symbol, and record the merge.
- Repeat until the vocabulary reaches the target size.
To tokenise new text, apply the learned merges in the same order. GPT-2 used about 50,257 tokens; recent models use 100,000 to 200,000. A useful rule of thumb for English with these tokenisers is roughly three-quarters of a word per token.
Why tokenisation leaks into behaviour
- Spelling and counting letters is hard because the model never sees letters. "strawberry" may be two or three tokens, and how many r's it contains is not visible in them.
- Arithmetic suffers when numbers are chopped inconsistently ("1234" might be "12" + "34" or "123" + "4"). Some models split digits individually for this reason.
- Languages underrepresented in the tokeniser's training data need more tokens per sentence, which makes them slower and more expensive to process (Petrov et al., 2023).
- A leading space is usually part of the token: " hello" and "hello" are different ids.
Text is split into subword tokens by BPE: repeatedly merge the most frequent adjacent pair. The model reasons over those tokens, not letters, which explains several of its odd failures.