Theory test: Language models

6 questions. Pass with 5 correct.

1. 01 Each step of training byte-pair encoding merges:

   Two random charactersThe most frequent adjacent pair of symbolsThe two least frequent tokensWhole words into sentences
2. 02 A model that spreads probability uniformly over a 100-token vocabulary has perplexity:

   110100$\log 100$
3. 03 As sampling temperature approaches 0, sampling becomes:

   Uniformly randomGreedy: always the most likely tokenImpossibleTop-p with p = 0.5
4. 04 Top-p sampling with $p=0.9$ samples from:

   The top 90 tokensThe smallest set of most-likely tokens whose probabilities sum to at least 0.9Tokens with probability above 0.9A random 90% of the vocabulary
5. 05 By the chain rule, $P(\text{the cat sat})$ equals:

   $P(\text{the})P(\text{cat})P(\text{sat})$$P(\text{the})P(\text{cat}\mid \text{the})P(\text{sat}\mid \text{the cat})$$P(\text{sat}\mid \text{the cat})$ only$P(\text{the}\mid \text{cat sat})$
6. 06 Why do LLMs often miscount the letters in a word?

   They were never trained on spellingThey see subword tokens, not individual charactersTheir vocabulary has no lettersTemperature is too high