GRPO, Step by Step

Group Relative Policy Optimization (GRPO) trains a model by comparing several answers to the same prompt. It scores the answers, measures how each compares with the group, and updates the model toward the better-scoring responses.

The distinctive choice is the baseline. GRPO uses the group’s rewards instead of training a separate value model, often called a critic. The DeepSeekMath paper introduced the method as a way to reduce the memory burden of reinforcement learning for language models.

This walkthrough uses three answers and small numbers to explain the calculation. The Python example computes a simplified objective; it is not a language-model trainer.

What GRPO changes

In reinforcement learning, a policy is the model that generates an answer. A reward function scores the result. Training changes the policy to favor outputs that score better.

PPO-based language-model training commonly uses a learned value function to estimate a baseline for those rewards. GRPO replaces that estimate with statistics from a group of sampled answers. It still needs generation, scoring, and policy updates, so removing the critic does not make training free.

When the approach is useful

The approach is attractive when you can generate several responses to a prompt and score them meaningfully. Mathematical answers and executable code offer relatively direct checks. Open-ended tasks require more judgment about what the reward actually measures.

Choosing a useful scoring rule is also the starting point for the evaluation checks in LLM-Expect. Those checks evaluate a function; they do not train it.

A poor reward function remains a poor reward function with GRPO. The method will optimize the signal you give it, including shortcuts you did not intend.

The calculation

1. Sample several answers

Suppose the prompt is: “A car travels 120 miles in two hours. What is its average speed?”

The sampling policy produces:

“The speed is 60 mph.”
“60 mph.”
“The car goes 50 mph.”

The group gives us a comparison for this specific question. Sampling more responses costs more compute, but can provide more varied candidates.

2. Assign rewards

For this numerical example, use these illustrative scores:

Answer	Reward
The speed is 60 mph.	0.9
60 mph.	0.7
The car goes 50 mph.	0.4

These scores are chosen to demonstrate the arithmetic. They are not a recommended rubric. If correctness is the only goal, the first two answers deserve the same reward and the wrong answer should receive none. Rewarding a longer answer simply because it is longer can teach verbosity.

A reward may be binary, continuous, or a weighted combination of checks. Decide what each component measures and inspect examples where they disagree. A readable wrong answer should not outrank a terse correct one unless your task has an explicit reason for that choice.

3. Calculate relative advantages

An advantage describes how an answer’s reward compares with the rest of its group:

advantage_i = (reward_i - mean(group_rewards)) / std(group_rewards)

For rewards [0.9, 0.7, 0.4], using the population standard deviation:

mean = (0.9 + 0.7 + 0.4) / 3 = 0.666667

std = sqrt(((0.9 - mean)^2 + (0.7 - mean)^2
            + (0.4 - mean)^2) / 3)
    = 0.205480

Answer	Reward	Advantage
The speed is 60 mph.	0.9	1.135550
60 mph.	0.7	0.162221
The car goes 50 mph.	0.4	-1.297771

A positive advantage means above average within this group. It does not prove the answer is good in absolute terms: the best of several wrong answers can still have a positive advantage.

If all rewards are equal, the standard deviation is zero. An implementation must handle that case, for example by returning zero advantages. There is no within-group reward difference to learn from.

4. Compare current and old probabilities

The policy ratio measures how the probability assigned to a sampled choice changes during optimization:

ratio = current_policy_probability / old_policy_probability

The old policy generated the samples. The current policy is being updated. In the actual language-model objective, these ratios are calculated per token, conditioned on the prompt and preceding tokens.

To keep the arithmetic small, this example treats each complete answer as one action in a toy three-action policy:

Answer	Current probability	Old probability	Ratio
The speed is 60 mph.	0.35	0.30	1.166667
60 mph.	0.45	0.40	1.125000
The car goes 50 mph.	0.20	0.30	0.666667

A ratio above one means the current policy assigns the action more probability than the old policy did. It is not a calibrated measure of confidence or correctness.

Clipping provides a bounded version of the ratio:

clipped_ratio = clip(ratio, 1 - epsilon, 1 + epsilon)

With epsilon = 0.2, the clipped interval is [0.8, 1.2]. The first two ratios stay unchanged; the third clips to 0.8. Clipping changes the optimization incentive. It does not impose a hard limit on every probability update or guarantee stable training.

5. Apply the clipped surrogate

The contribution is the smaller of two products:

surrogate_i = min(ratio_i * advantage_i,
                  clipped_ratio_i * advantage_i)

Multiply before taking the minimum. Writing min(ratio, clipped_ratio) * advantage gives the wrong result when the advantage is negative.

For the third answer:

unclipped = 0.666667 * -1.297771 ≈ -0.865181
clipped   = 0.800000 * -1.297771 ≈ -1.038217
minimum   ≈ -1.038217

The contributions, calculated before rounding, are:

Answer	Surrogate contribution
The speed is 60 mph.	1.324808
60 mph.	0.182499
The car goes 50 mph.	-1.038217

For a positive advantage, the clipped objective stops rewarding an increase beyond the upper clipping bound. For a negative advantage, it stops rewarding a decrease below the lower bound. The sign matters to the calculation and to its interpretation.

6. Include the reference-policy penalty

The original GRPO formulation also penalizes divergence from a reference policy. That reference provides an anchor; it is distinct in purpose from the old policy used to generate the current batch.

For this toy example, write:

objective = mean(surrogate_contributions) - beta * KL

Using an illustrative KL = 0.05 and beta = 0.1:

mean contribution ≈ 0.156363
penalty           = 0.005
objective         ≈ 0.151363

The KL value here is supplied for illustration, not derived from the probability table. A real implementation estimates the penalty using the current and reference policies. The full objective averages token contributions within responses and across sampled groups; the paper’s equations 3–5 give that formulation.

This objective guides training. Its value alone is not a product-quality score, and values from different reward schemes or batches are not automatically comparable.

Check the arithmetic in Python

This example uses only the standard library:

from statistics import mean, pstdev

rewards = [0.9, 0.7, 0.4]
current_probs = [0.35, 0.45, 0.20]
old_probs = [0.30, 0.40, 0.30]
epsilon = 0.2
beta = 0.1
illustrative_kl = 0.05

def group_advantages(scores):
    center, spread = mean(scores), pstdev(scores)
    if spread == 0:
        return [0.0] * len(scores)
    return [(score - center) / spread for score in scores]

advantages = group_advantages(rewards)
ratios = [new / old for new, old in zip(current_probs, old_probs)]
clipped = [min(max(ratio, 1 - epsilon), 1 + epsilon) for ratio in ratios]
terms = [
    min(ratio * advantage, bound * advantage)
    for ratio, bound, advantage in zip(ratios, clipped, advantages)
]
objective = mean(terms) - beta * illustrative_kl

print("Advantages:", [round(value, 6) for value in advantages])
print("Contributions:", [round(value, 6) for value in terms])
print("Toy objective:", round(objective, 6))

Expected output:

Advantages: [1.13555, 0.162221, -1.297771]
Contributions: [1.324808, 0.182499, -1.038217]
Toy objective: 0.151363

There is no model, gradient update, tokenizer, or reward service in this example. Use it to understand the objective, not as a substitute for a training implementation.

The benefit and its limits

Removing the learned critic reduces one source of memory and training overhead. The total saving depends on model size, sampling, reward computation, and the implementation. It is not a fixed percentage.

GRPO also does not guarantee accurate answers, prevent overfitting, or automatically produce reasoning. Those outcomes depend on the model, tasks, rewards, data, and training process.

The product decision

Before considering reinforcement learning, establish that the task needs it. A better prompt, retrieval system, conventional rule, or supervised fine-tuning dataset may address the problem more directly.

If GRPO is justified, the product work is largely about the reward and the evaluation:

What behavior should improve, and how can it be scored?
Can a model earn reward while producing an answer users would reject?
What quality, latency, and cost tradeoffs are acceptable?
Which held-out tasks will show whether the improvement generalizes?

Where it fits in training

Start with a capable policy and task prompts. Supervised fine-tuning may establish useful behavior before reinforcement learning, depending on the training plan.

Generate response groups, score them, compute advantages, and optimize the policy. Evaluate on unseen tasks and inspect failure cases throughout. After deployment, monitoring may reveal a need to change the data or reward and run another training cycle. GRPO does not automatically learn from production feedback.

What to take away

GRPO’s useful simplification is replacing a learned value baseline with comparisons among answers to the same question. Understanding that comparison, the clipping calculation, and the limits of the reward tells you more than claims that the method guarantees cheaper or better AI.