Key summary

RLHF first trains a reward model on human comparison data where people pick the better of two responses, and then uses reinforcement learning (PPO) to tune the language model toward higher reward scores while applying a KL penalty so it does not drift too far from the reference model. DPO is an alternative that removes the separate reward model and reinforcement-learning loop, training the model directly on the same preference pairs.

RLHF (Reinforcement Learning from Human Feedback) is a core alignment technique that adjusts a language model beyond simple next-token prediction so its outputs follow human preferences. Based on the general flow described in the original papers, this post walks through how a reward model is trained, how PPO and the KL penalty update the policy, what reward hacking is, and how DPO differs.

What is RLHF?

One-line answer: RLHF (Reinforcement Learning from Human Feedback) is a training method that builds a reward model from human preference comparisons and uses reinforcement learning to align a language model’s outputs with human preferences.

The general flow described in the paper by Ouyang et al., “Training language models to follow instructions with human feedback” (2022), consists of three stages: (a) supervised fine-tuning (SFT) on demonstrations, (b) having people compare or rank several model outputs for the same prompt and training a reward model on these comparisons to give the preferred output a higher score, and (c) optimizing the fine-tuned model (the policy) against the reward model with the PPO reinforcement-learning algorithm, using a per-token KL penalty against the SFT (reference) model to keep the policy from drifting too far.

For an overview of the three-stage pipeline across pretraining, SFT, and RLHF, see Pretraining vs Fine-Tuning vs RLHF: What Stages Build an LLM?.

How is the reward model trained?

One-line answer: Given multiple responses to the same prompt, a person picks the better one, and the reward model learns from these relative comparisons to assign a higher score to the preferred response.

A reward model takes a prompt and a model response as input and outputs a scalar score. During training, the model generates several outputs for the same prompt, and human labelers compare or rank them to indicate which output is better.

Crucially, the reward model learns from relative comparisons (response A is better than response B) rather than from absolute numerical scores assigned directly by people. Across these comparison pairs, the reward model updates its weights so that the human-preferred response receives a higher score than the rejected one.

What does PPO change?

One-line answer: PPO updates the language model (policy) weights to achieve a higher score from the reward model while applying a per-token KL penalty so the policy does not drift too far from the reference model.

Once the reward model is trained, the fine-tuned language model acts as the policy in a PPO (Proximal Policy Optimization) reinforcement-learning loop:

  1. The policy generates a response to a prompt.
  2. The reward model scores the generated response.
  3. PPO updates the policy weights to increase the reward score.

Without a constraint, optimizing solely for the reward model score can cause the policy to drift into strange wording or exploit weaknesses of the reward model to get a high score (reward over-optimization). To prevent this, a per-token KL penalty against the SFT (reference) model is subtracted from the objective, keeping the updated policy close to the reference model.

RLHF loop: policy generates a response, reward model scores it, PPO updates weights, KL penalty keeps the policy near the reference model

What is reward hacking?

One-line answer: Reward hacking occurs when the policy exploits weaknesses in the reward model to drive up its score without actually producing better responses.

Because the reward model is only an imperfect proxy trained on finite human comparisons, the policy can find gaps in the reward model during optimization and raise its score without genuinely improving response quality. This phenomenon is known as reward hacking.

The idea that a high model score does not automatically mean a response is correct also connects to confidence calibration, discussed in What is an LLM confidence score? Why you should not just ask the model how sure it is, and what calibration means.

How do RLHF and DPO differ?

One-line answer: RLHF uses a separate reward model and a PPO reinforcement-learning loop, whereas DPO trains the language model directly on preference pairs using a classification-style loss.

As described in Rafailov et al., “Direct Preference Optimization: Your Language Model is Secretly a Reward Model” (2023), DPO trains the model directly on the same preference pairs with a simple classification-style loss, without fitting a separate reward model and without a reinforcement-learning loop. The DPO loss is a kind of classification loss, which relates to the topic What is cross-entropy loss? Why classification models use it instead of MSE.

AspectRLHF (Reward Model + PPO)DPO
What is neededSeparate reward model, reinforcement-learning loop, and reference modelPreference pairs and reference model only (no separate reward model)
Training methodPPO reinforcement learning against scores from the reward modelDirect optimization on preference pairs with a classification-style loss
Pros & consFlexible, but involves more moving parts and tuning overheadSimpler and more stable to run, but tied to the fixed preference dataset

RLHF (reward model + PPO) vs DPO

What are the frequently asked questions?

One-line answer: Here are concise answers on how RLHF relates to factual accuracy, why SFT alone is not enough, and follow-up methods that reduce human labeling.

Does RLHF guarantee higher factual accuracy?

No. RLHF reflects human preferences; it does not by itself guarantee factual accuracy. For more on hallucination limits and retrieval structures, see Day5–7: AI Limitations, Embedding and RAG Architecture, AI Agents and Tool Calling.

Is supervised fine-tuning (SFT) alone not enough?

SFT trains the model to imitate given demonstration examples, whereas comparisons between outputs capture “which response is better” when it is hard to write a single ideal demonstration.

What are RLAIF and Constitutional approaches?

They are approaches that use AI feedback in place of some human feedback when collecting preference labels.

One-line answer: Check out these related posts covering LLM training stages, attention mechanisms, hallucination limits, and confidence calibration.

What are the sources?

One-line answer: Below are the two original papers on RLHF and DPO referenced in this post.