Nestor G Pestelos Jr · Reference · Print this page

Artificial Intelligence · Model Post-Training

Reinforcement Learning with Verifiable Rewards (RLVR)

Reference entry · last updated September 10, 2026

Reinforcement learning with verifiable rewards (RLVR) is a post-training method that optimizes a language model policy against a reward computed by a deterministic checker rather than a learned reward model. The reward is usually binary: a completion scores 1 when an automatic verifier confirms it is correct, such as an exact-match answer key for mathematics, a unit-test suite for code, or a proof checker for formal logic, and 0 otherwise. It is the training signal behind the 2025 generation of reasoning models, and is also written "reinforcement learning from verifiable rewards."

1. First Principles: Verification as the Reward

RLVR applies to the subset of tasks where correctness is decidable by a program and the correct answer is already known for each training prompt. On that subset the reward function needs no modeling. For a prompt \(x\) with reference answer \(a_x\), the reward for a sampled completion \(y\) is the output of a verifier \(V\):[4, 5]

The policy is then optimized with the same KL-regularized reinforcement learning objective used in RLHF, with the learned reward model replaced by this rule:

Removing the learned reward model removes the failure surface it introduces. There is no reward-model overoptimization in the usual sense, because on the covered prompts the reward is ground truth rather than a fitted approximation of it. Preference data collection, rater disagreement, and the Bradley-Terry modeling assumption all drop out.

Two constraints define the method's reach. Coverage: a prompt is only usable if a verifier and a reference answer exist for it, which excludes open-ended writing, judgment, and most dialogue. Specification: the verifier defines "correct," so any gap between the verifier and true correctness is a gap the policy can learn to exploit. Goodhart's law returns at the level of the checker rather than a reward model.

2. The Training Loop

The pipeline that produced DeepSeekMath and the DeepSeek-R1 reasoning models runs a single reinforcement learning stage after supervised fine-tuning, or in the R1-Zero case directly on the base model.[4, 6]

2.1 Prompt Set with Known Answers

Training data is a set of prompts paired with reference answers, not demonstrations of the full solution. Common sources are math datasets with final-answer keys such as GSM8K and MATH,[1, 2] competitive-programming problems with test suites, and formal-theorem statements with machine-checkable proofs. The model must produce the reasoning itself; only the endpoint is graded.

2.2 The Verifier and the Reward

For each prompt the policy samples one or more completions, and each completion is scored by the verifier. Verifiers are domain-specific: symbolic equivalence checking against the answer key for mathematics, execution against unit tests for code, a proof assistant for formal logic. Implementations frequently split the reward into an accuracy term and a format term, for example rewarding a required answer delimiter or a reasoning block, so that the total is

The format term is itself a small rule-based specification and a common target for reward hacking.

2.3 Policy Update with GRPO

The policy gradient step most associated with RLVR is Group Relative Policy Optimization (GRPO), introduced with DeepSeekMath as a variant of PPO that removes the value network.[4, 7] For each prompt \(q\), GRPO samples a group of \(G\) completions \(\{o_1, \dots, o_G\}\), scores each with the verifier to get rewards \(\{r_1, \dots, r_G\}\), and standardizes within the group to form the advantage:

The group mean replaces the learned baseline, so completions are rewarded for beating their siblings on the same prompt. The update uses a PPO-style clipped ratio; see reinforcement learning for that objective in full. PPO with a value model is also used for RLVR; GRPO's appeal is lower memory during long-context reasoning rollouts.

3. Relationship to RLHF and Process Supervision

RLVR and RLHF share the reinforcement learning objective and differ in where the reward comes from. RLHF fits a reward model to human preference comparisons and optimizes against it;[8] RLVR replaces that model with a rule.

The two are complementary rather than exclusive. A production post-training recipe such as Tulu 3 runs supervised fine-tuning, then preference optimization, then RLVR as a later stage aimed at the verifiable skills.[5]

Along a second axis, RLVR as usually practiced is outcome supervision: it grades the final answer and says nothing about the steps. Process supervision instead scores each reasoning step, which Lightman et al. found trains a more reliable reward model on the MATH benchmark than outcome supervision does.[3] Process rewards require step-level labels or a trained process reward model and lose the "no learned reward model" property that makes outcome RLVR simple.

4. Verifiable Domains and Their Checkers

• Mathematics. Final-answer comparison with symbolic normalization to accept equivalent forms. The dominant testbed; also the domain where false negatives from imperfect equivalence checking are most reported.
• Code. Compilation plus a unit-test suite, with the pass rate or a pass/fail bit as reward. The verifier is only as complete as the tests.
• Formal proofs. A proof assistant such as Lean accepts or rejects a candidate proof. The strongest form of verification, with the narrowest data supply.
• Constrained instruction following. Instructions with checkable constraints, such as a word count, a required keyword, or valid JSON, admit a rule-based check.[5]

The common thread is a cheap, deterministic, low-false-positive check. Where a check is expensive, stochastic, or easily satisfied by a wrong answer, RLVR degrades toward the reward-hacking behavior it is meant to avoid.

5. Empirical Results

DeepSeekMath applied GRPO with rule-based math rewards and raised a 7B model's MATH accuracy well above its supervised baseline, establishing the recipe.[4] Tulu 3 reported that adding RLVR as a final stage improved targeted skills such as mathematics and precise instruction following without a general regression.[5] DeepSeek-R1 scaled the approach: an RL-only variant, R1-Zero, was trained from the base model with verifiable rewards and no supervised reasoning traces, and developed long chains of thought, self-checking, and backtracking on its own, though with readability and language-mixing problems that a supervised cold-start stage then addressed.[6]

6. Limitations and Open Questions

• Domain ceiling. RLVR trains only what a verifier can score. It adds no signal for style, helpfulness, or safety, so it supplements preference tuning rather than replacing it.
• Capability extension versus sampling. Yue et al. found that across large sampling budgets, base models reach a higher pass@k than their RLVR-tuned versions, and argue RLVR mostly reweights toward solutions the base model could already produce rather than adding new reasoning ability.[9]
• Reward-signal attribution. Shao et al. showed that random or spurious rewards produce large math gains for Qwen models by eliciting pretrained behavior, while the same signals barely help Llama3 or OLMo2, which complicates claims that a specific reward drove a specific gain.[10]
• Verifier gaming. The policy can learn to satisfy the checker without solving the task, for example matching an answer format while reasoning incorrectly, or exploiting weak unit tests. See reward hacking.
• Verifier coverage. Equivalence checkers reject valid alternative answers (false negatives) and weak checkers accept wrong ones (false positives). Both inject label noise the policy optimizes against.

7. Development

• 2021. Cobbe et al. introduce GSM8K and train a separate verifier to rank sampled math solutions;[1] Hendrycks et al. release the MATH dataset.[2]
• 2023. Lightman et al. compare process and outcome supervision on MATH and release step-level labels.[3]
• 2024. DeepSeekMath introduces GRPO with rule-based math rewards;[4] Tulu 3 names the method "Reinforcement Learning with Verifiable Rewards" and ships it as a post-training stage.[5]
• 2025. DeepSeek-R1 and R1-Zero demonstrate reasoning trained largely or entirely by verifiable rewards;[6] Yue et al. and Shao et al. publish limiting analyses.[9, 10]