Nestor G Pestelos Jr · Reference · Print this page

Artificial Intelligence · Reinforcement Learning

Group Relative Policy Optimization (GRPO)

Reference entry · last updated September 10, 2026

Group Relative Policy Optimization (GRPO) is a policy-gradient algorithm for language model post-training that removes the value network used by Proximal Policy Optimization. In place of a learned baseline, it samples a group of completions for each prompt, scores them, and measures each completion against the group's mean reward. It was introduced with DeepSeekMath in 2024 and became the standard optimizer for reasoning-model training after DeepSeek-R1.

1. First Principles: Baselines in Policy-Gradient Estimation

Policy-gradient methods estimate the gradient of expected return by weighting the score function of each action by a measure of how good that action was:[4, 5]

Any baseline \(b(s)\) that does not depend on the action leaves the gradient unbiased while changing its variance. A good baseline is close to the expected return from \(s\), so the bracket keeps only the part of the return attributable to the action choice. The estimator is otherwise dominated by the overall difficulty of the state.

Proximal Policy Optimization (PPO) supplies this baseline with a learned value function \(V_\phi(s)\), trained alongside the policy and combined with the observed rewards through generalized advantage estimation.[2, 3] For a large language model the value network is a second model of comparable size, roughly doubling the memory and adding its own training instability.

GRPO removes that network.[1] It exploits a structural feature of language model reinforcement learning: many completions can be sampled cheaply for the same prompt. The mean reward of a sampled group is itself an estimate of the expected return for that prompt, so it can serve as the baseline directly. No value function is fitted, and the baseline is exact for the group rather than a generalization from other prompts.

2. The GRPO Objective

2.1 Group Sampling and the Relative Advantage

For each question \(q\), GRPO samples a group of \(G\) completions \(\{o_1, \dots, o_G\}\) from the current policy (DeepSeekMath used \(G = 64\)) and scores each with the reward function.[1] For outcome supervision, where only the final result is scored, every token of completion \(o_i\) receives the same advantage, computed by standardizing the group's rewards \(\mathbf{r} = \{r_1, \dots, r_G\}\):

The mean term is the group-relative baseline. The division by the group standard deviation rescales the update so that questions of different reward spread contribute on a common scale. A group whose completions all receive the same reward has zero standard deviation and produces no gradient.

2.2 The Clipped Surrogate and the KL Term

The policy is updated with a PPO-style clipped objective over per-token probability ratios, averaged over the tokens of each completion and then over the group:[1, 2]

with the token-level ratio

Two choices distinguish this from the PPO setup used in RLHF. The clipping range \(\varepsilon\) bounds each update, as in PPO. The reference-policy penalty is added directly to the loss as a separate term rather than folded into the reward, which keeps the advantage estimate uncomplicated.[1] GRPO estimates that penalty with an unbiased, always-positive estimator:[1]

2.3 Process-Supervision Variant

When a reward is available for intermediate steps rather than only the final answer, GRPO assigns each token the normalized sum of the step rewards that follow it within the same completion, so that advantages vary along the sequence.[1] This keeps the group-relative baseline while allowing credit to be placed on specific reasoning steps.

3. Comparison with PPO and REINFORCE-Style Methods

GRPO sits between PPO and the older REINFORCE family. All three optimize the same policy-gradient objective and differ in how they estimate the baseline.

RLOO is the closest relative. Ahmadian et al. argued that for language model alignment the full PPO machinery is unnecessary and that a leave-one-out baseline over a small group recovers most of the benefit at lower cost.[8] GRPO differs mainly in using the plain group mean rather than a leave-one-out mean, and in the standard-deviation rescaling, which is the part later work questions.

4. Why GRPO Spread: Verifiable Rewards and Reasoning Models

GRPO was designed for mathematical reasoning, where the reward comes from a deterministic checker rather than a reward model. See reinforcement learning with verifiable rewards. That setting suits the algorithm: sampling a large group is inexpensive, the reward scale is stable across prompts, and there is no learned reward model whose errors a value network would have to track.

DeepSeek-R1 used GRPO to train a frontier reasoning model largely through reinforcement learning, including an RL-only variant trained from the base model with no supervised reasoning traces.[7] After that result, GRPO and its descendants became the common choice for open reasoning-model training, and most later work frames its contribution as a modification of GRPO.

5. Known Biases and Variants

Two normalizations in the GRPO objective have been identified as sources of bias, and later systems adjust them.

• Dr. GRPO (Liu et al., 2025). The per-completion \(1/|o_i|\) averaging gives long completions a smaller per-token gradient, which the authors show pushes the policy to make incorrect responses longer during training. Their variant removes the length and standard-deviation normalizers, reporting similar reasoning accuracy with shorter outputs.[9]
• DAPO (Yu et al., 2025). An open large-scale system built on four changes to GRPO: Clip-Higher, which decouples the lower and upper clipping bounds to limit entropy collapse; Dynamic Sampling, which discards prompts whose completions are all correct or all incorrect so every batch item carries a gradient; a token-level policy-gradient loss in place of the per-completion average; and overlong reward shaping, a graded penalty for responses near the length limit.[10]

Both analyses also target the same degenerate case: a prompt whose completions all score identically produces no gradient and wastes its samples, which is what DAPO's dynamic sampling removes.

6. Development

• 1992. Williams formalizes REINFORCE, the score-function policy gradient with an optional baseline.[4]
• 2015 to 2017. Generalized advantage estimation[3] and then PPO[2] make actor-critic policy gradients stable enough for routine use.
• 2022. PPO with a learned value model and a KL-shaped reward becomes the standard RLHF optimizer.[6]
• 2024. DeepSeekMath introduces GRPO, replacing the value model with a group-relative baseline.[1] Ahmadian et al. make a parallel case for REINFORCE-style baselines such as RLOO.[8]
• 2025. DeepSeek-R1 scales GRPO to a frontier reasoning model;[7] Dr. GRPO and DAPO correct its normalization biases.[9, 10]