Theory test: Large language models

6 questions. Pass with 5 correct.

1. 01 By the Chinchilla rule of thumb (about 20 tokens per parameter), a compute-optimal 70B-parameter model trains on roughly:

   70 billion tokens1.4 trillion tokens20 billion tokens70 trillion tokens
2. 02 The reward model in RLHF is trained on:

   The pretraining corpusHuman comparisons between pairs of responsesThe test setRandomly generated rewards
3. 03 What does the KL penalty in RLHF do?

   Speeds up trainingKeeps the policy close to the SFT model so it can't drift into exploiting the reward modelMakes responses shorterRemoves the need for a reward model
4. 04 Under the Bradley–Terry model, $P(A\succ B)$ is:

   $r(A)/(r(A)+r(B))$$\sigma (r(A)-r(B))$$r(A)-r(B)$$\sigma (r(A))\sigma (r(B))$
5. 05 Why do language models hallucinate?

   They run out of memoryTraining rewards plausible continuations, not verified truthTemperature is always too highTheir tokenizer is broken
6. 06 Retrieval-augmented generation:

   Retrains the model on new documentsRetrieves relevant documents and puts them into the model's contextRemoves the context window limitOnly works with images