Nestor G Pestelos Jr · Reference · Print this page

Archived version captured 20260911. Read the current entry.

Artificial Intelligence · Model Alignment

Reinforcement Learning from Human Feedback (RLHF), archived 20260911

Reference entry · last updated September 10, 2026

Reinforcement learning from human feedback (RLHF) is a machine learning method that fits a reward model to human comparisons of candidate outputs, then optimizes a generative policy against that learned reward while a divergence penalty holds the policy near its starting point. It is the dominant post-training step for aligning large language models with instructions, tone, and safety norms that are simple to judge by example but hard to write down as a rule.

1. First Principles: Preference as the Training Signal

RLHF exists to optimize an objective that cannot be written directly. Qualities such as helpfulness, harmlessness, and honesty have no closed-form scoring function, but a person shown two responses to the same prompt can usually say which one is better. RLHF converts that comparative judgment into a scalar reward and then into gradient updates.[1]

The method assumes a latent reward function \(r^*(x, y)\) over prompt \(x\) and response \(y\), and models a human's pairwise choice with the Bradley-Terry model: given a preferred response \(y_w\) and a rejected response \(y_l\), the probability that a rater picks \(y_w\) is[2, 8]

where \(\sigma\) is the logistic function. Only reward differences are identifiable, so an additive constant is free, and the model treats disagreement between raters as logistic noise around a single shared utility.

Given a reward function, the alignment target is the Kullback-Leibler (KL) constrained policy objective. The tuned policy \(\pi_\theta\) should raise expected reward without moving far from a reference policy \(\pi_{\text{ref}}\), usually the supervised fine-tuned model:[4]

The coefficient \(\beta\) sets the price of divergence. A small \(\beta\) lets the policy chase reward and drift into degenerate text; a large \(\beta\) keeps fluency and diversity but limits how much behavior can change. RLHF therefore needs three inputs: a competent base policy, a dataset of human comparisons, and a divergence budget.

2. The Standard Three-Stage Pipeline

The pipeline that produced InstructGPT and most instruction-tuned chat models runs in three stages: supervised fine-tuning, reward model training, and reinforcement learning against that reward.[4]

2.1 Supervised Fine-Tuning

A base model is first fine-tuned on curated prompt-response demonstrations so that it follows the conversational schema and produces on-task answers. This model becomes both the starting point for reinforcement learning and the reference policy \(\pi_{\text{ref}}\) in the KL term.[4]

2.2 Reward Modeling

Human labelers rank or compare multiple sampled responses to each prompt. A reward model \(r_\psi(x, y)\), typically the SFT model with a scalar output head, is trained on these comparisons by minimizing the negative log-likelihood of the Bradley-Terry model:[3, 4]

The reward model compresses a finite set of human comparisons into a function that can score any response. Its accuracy on held-out comparisons, and its behavior far from the data it was trained on, bound the quality of everything downstream.

2.3 Policy Optimization with PPO

The policy is then optimized against the reward model with a policy gradient algorithm, most often Proximal Policy Optimization (PPO).[6] The KL constraint is applied as a per-token penalty folded into the reward, so the quantity actually maximized for a full response is

Token generation is treated as a sequential decision process: the state is the prompt plus the tokens produced so far, each action is the next token, and the environment transition is deterministic concatenation. PPO's clipped surrogate objective limits how far each update moves the policy; see reinforcement learning for that objective in full. A separate value network estimates the advantage, which adds memory and tuning cost and motivates the offline alternatives below.

3. Direct Preference Optimization

Direct Preference Optimization (DPO) removes the separate reward model and the online sampling loop.[8] Rafailov et al. showed that the optimal policy for the KL-constrained objective corresponds to an implicit reward

where \(Z(x)\) is a partition function that cancels inside a Bradley-Terry difference. Substituting this parameterization into the preference loss gives an objective in the policy alone, trained on a fixed dataset of comparison pairs with a binary cross-entropy loss:

DPO reaches alignment quality comparable to PPO-based RLHF on many benchmarks with a simpler and more stable training loop. Because it only sees a fixed preference set, it cannot gather fresh comparisons on its own current outputs, which online RLHF can.

4. Reward Model Overoptimization

The reward model is a proxy for human judgment, not human judgment itself. Optimizing against it past a point makes the proxy score keep rising while the true quality it stands in for stalls or falls, an instance of Goodhart's law. Gao et al. measured this by holding out a larger "gold" reward model as a stand-in for ground truth and tuning against a smaller proxy.[9] They fit simple functional forms relating the gold-reward gain to the square root of the KL divergence between the tuned and initial policies, with coefficients that vary predictably with reward-model size and the amount of preference data. Larger reward models and more comparisons push the turning point further out but do not remove it. The KL penalty, early stopping on a held-out judge, and reward-model ensembles are the common mitigations.

5. Reinforcement Learning from AI Feedback

Human comparison labeling is the cost and latency bottleneck of RLHF. Reinforcement learning from AI feedback (RLAIF) replaces some or all of the human labels with preferences generated by another language model.[10, 11]

Constitutional AI applies this to harmlessness: the model critiques and revises its own responses against a written list of principles, then a preference model is trained on AI-labeled comparisons and used for the RL stage, keeping human input mostly at the level of the principles.[10] Lee et al. compared RLAIF against RLHF on summarization and dialogue and reported comparable human win rates, along with a variant that scores responses directly with an off-the-shelf model instead of training a separate reward model.[11] AI feedback inherits the biases and blind spots of the labeling model.

6. Limitations and Open Problems

Casper et al. survey the failure modes of RLHF and separate problems that better engineering can address from ones that are structural to the approach.[12] The recurring issues:

• Feedback quality. Raters disagree, make errors, apply inconsistent standards, and can be misled by fluent or confident-sounding answers. Labeler pools are small and not representative of the eventual user base.
• Reward misspecification. A single scalar cannot capture plural, context-dependent human values, and the Bradley-Terry model assumes a consistency that real preferences lack.
• Sycophancy. Because the reward reflects what raters approved of, the policy learns to favor agreeable and flattering responses over accurate ones. See sycophancy.
• Distribution shift and reward hacking. The policy is optimized on its own shifting output distribution, which pulls it toward regions where the reward model is unreliable and exploitable. See reward hacking.
• Diversity loss. Preference optimization narrows the output distribution, reducing calibration and generation variety, part of the alignment tax.
• Systemic cost. The full pipeline needs demonstration data, a labeling operation, a trained reward model, and a stable distributed RL setup, which concentrates the method among well-resourced labs.

7. Development

• 2017. Christiano et al. train deep RL agents on simulated robotics and Atari from human comparisons of short trajectory clips, with humans labeling under one percent of interactions.[1]
• 2019. Ziegler et al. apply preference-based reward learning with a KL penalty to language models, fine-tuning GPT-2 for continuation style and summarization.[5]
• 2020. Stiennon et al. use RLHF for summarization, with tuned summaries preferred to human reference summaries on Reddit TL;DR.[7]
• 2022. Ouyang et al. put RLHF at the center of instruction following; outputs from a 1.3B InstructGPT model are preferred to those of 175B GPT-3.[4] Bai et al. apply it to a combined helpful and harmless assistant with iterated online training and red-teaming.[3]
• 2022 to 2024. DPO removes the online RL loop,[8] Gao et al. quantify overoptimization,[9] and Constitutional AI and RLAIF shift much of the labeling to models.[10, 11]