What is RLHF and how does it make LLMs helpful?

RLHF (Reinforcement Learning from Human Feedback) is the training technique that turns a raw text predictor into a helpful, safe, instruction-following assistant by using human preference data as a reward signal.

AIMachine Learning

Overview

A base LLM trained only on next-token prediction is unpredictable. Ask it a question and it might continue the text in any direction rather than actually answering. RLHF is the additional training phase that transforms a raw language model into a helpful assistant. Human annotators rate outputs for helpfulness and safety, those ratings train a reward model, and the LLM is fine-tuned to maximise that reward.

Key Points

•RLHF has three phases: supervised fine-tuning on demonstrations, reward model training on human preference pairs, and RL optimisation with PPO.
•Step 1 - Supervised Fine-Tuning (SFT): the model is fine-tuned on thousands of high-quality human-written (prompt, ideal response) pairs.
•Step 2 - Reward Model (RM): human raters compare two model outputs and pick the better one. These preferences train a separate reward model.
•Step 3 - RL with PPO: the SFT model is fine-tuned to generate outputs that score high according to the reward model.
•Constitutional AI (Anthropic) and RLAIF are variants that reduce reliance on expensive human labelling by using AI feedback.
•RLHF is why Claude, ChatGPT, and Gemini answer questions helpfully rather than just predicting the next word of a document.

Examples

What raw base model vs RLHF model produce

text

Prompt: "What is the capital of France?"

Base model (raw next-token prediction):
  "What is the capital of France? The capital of France has been a subject of
   historical debate. France, officially the French Republic, is a country in..."
  (continues document-style, never answers directly)

RLHF model (instruction-tuned):
  "The capital of France is Paris."
  (directly answers the question)

RLHF training pipeline overview

python

# Phase 1: Supervised Fine-Tuning (SFT)
# Trainers write ideal responses to prompts
sft_data = [
  {"prompt": "Explain recursion simply.", "response": "Recursion is when a function calls itself..."},
  {"prompt": "Fix this Python bug.", "response": "The bug is on line 3: you used = instead of ==..."},
  # thousands more high-quality examples
]
sft_model = fine_tune(base_llm, sft_data)

# Phase 2: Reward Model training
# Human raters compare pairs: which response is better?
preference_data = [
  {"prompt": "...", "chosen": "better response", "rejected": "worse response"},
  # thousands of pairs
]
reward_model = train_reward_model(preference_data)

# Phase 3: PPO (Proximal Policy Optimization)
# Fine-tune the SFT model to generate responses the reward model scores highly
# while not drifting too far from the SFT model (the KL divergence penalty)
rlhf_model = ppo_train(
  policy_model=sft_model,
  reward_model=reward_model,
  kl_penalty=0.02,   # prevents reward hacking
)

This is a simplified conceptual overview. Real RLHF training involves millions of preference comparisons and weeks of GPU time.

Direct Preference Optimisation (DPO) - the simpler alternative

python

# DPO (2023) achieves similar results to RLHF without a separate reward model
# It directly optimises the policy using preference pairs

from trl import DPOTrainer, DPOConfig

trainer = DPOTrainer(
  model=sft_model,
  ref_model=sft_model_copy,   # frozen reference model
  args=DPOConfig(beta=0.1),   # beta controls deviation from reference
  train_dataset=preference_dataset,  # {"prompt", "chosen", "rejected"}
)

trainer.train()
# DPO is now the dominant approach for open-source model alignment

Pro Tip

Understanding RLHF helps you prompt models better. RLHF models are optimised to be helpful, so they tend to over-explain, add caveats, and hedge answers. If you want terse, direct output, explicitly instruct the model: "Be brief. No caveats. Answer in one sentence." It needs permission to break its trained-in verbosity.