Ask a base model "What is the capital of France?" and it might answer, or it might continue with more quiz questions, because on the web a list of questions is often followed by more questions. Turning a predictor into an assistant that follows instructions, admits uncertainty and refuses harmful requests takes further training. The landmark recipe is from OpenAI's InstructGPT paper (Ouyang et al., 2022):
1. Supervised fine-tuning (SFT)
Collect demonstrations: prompts paired with high-quality responses written by people. Continue training the base model on them with the same next-token loss. The model learns the format and tone of an assistant from relatively few examples (thousands to tens of thousands), because the knowledge is already there from pretraining.
2. A reward model from preferences
Writing ideal answers is hard; comparing two answers is easier. Show labellers a prompt with two model responses and ask which is better. Train a reward model (usually the language model with a scalar output head) to score responses so that preferred ones score higher. The standard Bradley–Terry model says the probability that response A beats B depends only on the difference of their rewards:
3. Reinforcement learning
Finally, fine-tune the model to produce responses the reward model scores highly, using a reinforcement learning algorithm (originally PPO). A penalty on the KL divergence from the SFT model keeps it from drifting into strange text that games the reward model, a failure called reward hacking: the reward model is only an imperfect proxy for what people actually want.
The InstructGPT result: labellers preferred outputs of the 1.3B-parameter RLHF model over those of the 175B GPT-3 base model. Alignment training changed how useful the model was more than a 100× increase in size did.
Newer variants
- Direct Preference Optimization (DPO) (Rafailov et al., 2023) shows the RLHF objective can be optimised directly on preference pairs with a simple classification-style loss, with no separate reward model or RL loop.
- Constitutional AI / RLAIF (Bai et al., 2022, Anthropic) uses a written set of principles and AI-generated feedback to supply many of the preference labels, reducing the need for human labelling of harmful content.
- Reinforcement learning from verifiable rewards: for maths and code, answers can be checked automatically, giving a reward signal that can't be flattered. Training on such rewards is behind recent "reasoning" models that write long chains of thought before answering.
Assistants are made in three steps: supervised fine-tuning on demonstrations, a reward model trained on human preferences (Bradley–Terry), and RL against that reward with a KL leash. DPO and AI feedback are simpler variants.