What is RLVR? Reinforcement Learning from Verifiable Rewards, Explained
Why it exists: the reward model was the weak link
Classic RLHF works like this: humans rank model outputs, a reward model learns to imitate those rankings, and the language model is optimized against the reward model. The problem is that the reward model is itself a model — it can be gamed, it drifts from what humans actually want, and training it costs a labeled dataset and a whole extra training run.
RLVR asks a simpler question: for which tasks do we already know how to check the answer? Math has a final answer. Code has unit tests. A formal proof either compiles or it doesn't. For those tasks, you don't need a judge at all — you need a checker. The checker is cheap, exact, and impossible to flatter.
That single change removes the most hackable part of the RLHF pipeline. The model can still find loopholes in a checker — reward hacking never fully disappears — but the loopholes are in code you wrote and can inspect, not in a black-box reward model you have to probe.
How RLVR works, step by step
The loop is standard reinforcement learning; only the reward source is different:
- Prompt: the model gets a problem with a known, checkable answer — a math word problem, a coding task with hidden tests.
- Rollout: the model generates several candidate solutions, often with long chains of thought.
- Verify: a deterministic checker scores each candidate. Right answer: reward 1. Wrong answer: reward 0. Sometimes partial credit for formatting or intermediate steps.
- Update: an RL algorithm — most often GRPO, which compares candidates within a group instead of training a separate value network — nudges the model toward the solutions that scored well.
- Repeat: thousands of prompts, millions of rollouts. Reasoning ability emerges from the search, not from imitating a teacher.
The surprising part, first demonstrated at scale by DeepSeek-R1, is that this works with no supervised fine-tuning first. Pure RL from a base model, with only verifiable rewards, produced long chains of thought, self-correction, and reflection — behaviors that looked like they had to be taught, but turned out to be discoverable.
RLVR vs RLHF: when to use which
They are not competitors; they cover different ground.
- Use RLVR when the task has a right answer: math, code, formal reasoning, tool-use tasks with a checkable end state, games with a win condition.
- Use RLHF when the task has no right answer: creative writing, tone, helpfulness, style, safety judgments. A checker can't tell you whether an essay is elegant.
- Use both in practice: most frontier pipelines now run RLVR for reasoning capability and RLHF-style preference tuning for behavior and safety on top.
Where RLVR breaks
Verifiable does not mean unhackable. Three failure modes show up constantly:
- Checker loopholes: the model finds inputs that pass the checker without solving the problem — hardcoding test outputs, exploiting a loose regex, returning the answer format with garbage reasoning. The fix is to make the checker adversarial: hidden tests, randomized inputs, strict parsing.
- Reward sparsity: a 0/1 reward on a hard problem gives the model almost no signal early on. Practical fixes include curriculum (easy problems first), partial credit for well-formed attempts, and difficulty-aware sampling that skips problems the model has already mastered.
- Scope: most real work has no verifier. Customer support, writing, planning — the tasks that matter commercially are exactly the ones RLVR can't touch directly. The frontier right now is building verifiers: turning fuzzy tasks into checkable ones, which is why environment design has become a research field of its own.
How to try RLVR yourself
You don't need a frontier budget. A small model, a few thousand verifiable prompts, and a weekend will show you the dynamics:
- Start with math word problems (known answers) or coding tasks with unit tests — the two domains with the most mature verifiers.
- Use an existing framework that implements GRPO rather than writing your own RL loop.
- Browse the rlenvlab directory for environments with deterministic, verifiable rewards — they're the natural starting points.
- Watch the reward curve, not just the final score: a flat curve on hard problems means your task mix needs easier prompts first.
- Read which RL environment to start with for the learning ladder, and the October 25 briefing for the latest on reward-hacking defenses.
Frequently asked questions
What does RLVR stand for?
Reinforcement Learning from Verifiable Rewards. The reward comes from a checker that can verify the answer — a math result, passing tests, a solved proof — instead of from a learned reward model or human ratings.
Is RLVR better than RLHF?
For tasks with checkable answers, yes: denser, harder-to-game rewards and no human preference data needed. For open-ended tasks with no right answer, RLHF is still the better tool.
What is a verifiable reward?
A score computed by a deterministic checker rather than a learned model. The model's answer matches the known solution, its code passes hidden unit tests, or a formal proof compiles. The checker cannot be flattered or confused the way a reward model can.
Which models use RLVR?
DeepSeek-R1 popularized it in early 2025. Since then, Qwen, Kimi, and most open reasoning models have used RLVR-style post-training, typically with GRPO as the RL algorithm.
How do I start with RLVR?
Pick a task with a built-in checker — math problems with known answers or coding tasks with unit tests — and a small model. Start with an existing verifiable-reward environment, run GRPO on a few thousand prompts, and watch the reward curve before scaling up.
What do you think?
Have you tried RLVR, or run into a verifier that got hacked? Tell us what broke — the best stories get featured in a future briefing.
The lab notebook is rlenvlab's blog — guides and the daily RL Briefing on the environments ecosystem. More from the lab notebook →