Introduction

Pretraining teaches a model to represent and continue the distribution of its training data. Post-training changes that distribution toward behavior we care about: following instructions, solving tasks, respecting preferences, using tools, or maximizing a measurable reward.

For language models, the object being trained is still a conditional probability distribution,

\[\pi_{\theta}(o_t \mid q, o_{<t}),\]

where $q$ is a prompt, $o_{<t}$ is the response prefix, and $o_t$ is the next token. What changes between supervised fine-tuning, PPO, and GRPO is primarily the coefficient placed in front of the familiar log-likelihood gradient

\[\nabla_{\theta}\log \pi_{\theta}(o_t \mid q,o_{<t}).\]

This gives a useful unifying picture:

  • Supervised fine-tuning increases the probability of selected demonstrations.
  • Policy gradients increase or decrease the probability of sampled actions according to their return.
  • PPO uses an advantage estimate and constrains how far the new policy can move from the policy that generated the data.
  • GRPO retains the PPO update but estimates the baseline by comparing several responses to the same prompt instead of training a critic.

The purpose of this primer is to derive that picture rather than state it. In particular, we will prove the causality step that justifies reward-to-go, derive baselines and advantages, and then build PPO and GRPO from those pieces.

Reinforcement Learning Notation

Consider an episodic Markov decision process. At step $t$, an agent observes a state $s_t$, samples an action $a_t$, receives a reward $r_t$, and moves to $s_{t+1}$. A trajectory is

\[\tau=(s_0,a_0,r_0,s_1,a_1,r_1,\ldots,s_T).\]

The policy is a probability distribution

\[\pi_{\theta}(a_t\mid s_t),\]

parameterized by $\theta$. For now, define the undiscounted trajectory return as

\[R(\tau)=\sum_{t=0}^{T-1}r_t.\]

The training objective is

\[J(\theta)=\mathbb{E}_{\tau\sim\pi_{\theta}}[R(\tau)].\]

For a language model, the translation is direct:

\[s_t=(q,o_{<t}),\qquad a_t=o_t.\]

The environment state is the prompt and generated prefix, and the action is the next token. The probability of a complete response factorizes autoregressively:

\[\pi_{\theta}(o\mid q) = \prod_{t=1}^{|o|}\pi_{\theta}(o_t\mid q,o_{<t}).\]

Deriving the Policy Gradient

The probability of a trajectory is

\[p_{\theta}(\tau) = p(s_0) \prod_{t=0}^{T-1} \pi_{\theta}(a_t\mid s_t) P(s_{t+1}\mid s_t,a_t).\]

The initial-state distribution and environment dynamics do not depend on the policy parameters. Starting from the objective,

\[J(\theta)=\sum_{\tau}p_{\theta}(\tau)R(\tau),\]

we differentiate:

\[\nabla_{\theta}J(\theta) = \sum_{\tau}\nabla_{\theta}p_{\theta}(\tau)R(\tau).\]

Use the log-derivative identity

\[\nabla_{\theta}p_{\theta}(\tau) = p_{\theta}(\tau)\nabla_{\theta}\log p_{\theta}(\tau).\]

Then

\[\nabla_{\theta}J(\theta) = \mathbb{E}_{\tau\sim\pi_{\theta}} \left[ R(\tau)\nabla_{\theta}\log p_{\theta}(\tau) \right].\]

Because only the policy factors depend on $\theta$,

\[\nabla_{\theta}\log p_{\theta}(\tau) = \sum_{t=0}^{T-1} \nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t).\]

Therefore,

\[\boxed{ \nabla_{\theta}J(\theta) = \mathbb{E} \left[ R(\tau) \sum_{t=0}^{T-1} \nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) \right]. }\]

This is the score-function or REINFORCE estimator [Wil92]. It is the central mechanism behind the methods discussed below. We do not differentiate through the sampled discrete action. Instead, we differentiate the log-probability of having sampled it.

Why Past Rewards Disappear

The previous formula multiplies every policy-score term by the entire trajectory return. At first this seems to imply that an action at time $10$ should be credited for a reward received at time $0$. Causally, that cannot be right: the later action cannot change an event that has already happened.

The mathematical statement is exactly

\[\mathbb{E} \left[ r_0\nabla_{\theta}\log\pi_{\theta}(a_{10}\mid s_{10}) \right] =0.\]

More generally, for every $k<t$,

\[\boxed{ \mathbb{E} \left[ r_k\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) \right] =0. }\]

Conditional-expectation proof

Let $H_t$ denote everything known immediately before sampling $a_t$:

\[H_t=(s_0,a_0,r_0,\ldots,s_t).\]

For $k<t$, the reward $r_k$ is already contained in $H_t$. It is therefore fixed after conditioning on $H_t$. By the law of total expectation,

\[\begin{aligned} &\mathbb{E} \left[ r_k\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) \right] \\ &=\mathbb{E} \left[ \mathbb{E} \left[ r_k\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) \mid H_t \right] \right] \\ &=\mathbb{E} \left[ r_k \mathbb{E} \left[ \nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) \mid H_t \right] \right]. \end{aligned}\]

Conditional on the history, the only remaining randomness in this expression is the newly sampled action

\[a_t\sim\pi_{\theta}(\cdot\mid s_t).\]

The expected score of a normalized probability distribution is zero:

\[\begin{aligned} \mathbb{E}_{a_t\sim\pi_{\theta}} \left[ \nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) \right] &= \sum_a \pi_{\theta}(a\mid s_t) \nabla_{\theta}\log\pi_{\theta}(a\mid s_t) \\ &= \sum_a\nabla_{\theta}\pi_{\theta}(a\mid s_t) \\ &= \nabla_{\theta}\sum_a\pi_{\theta}(a\mid s_t) \\ &= \nabla_{\theta}1 =0. \end{aligned}\]

Substituting this into the conditional expectation proves the claim.

A subtle point is worth emphasizing. The past reward $r_k$ can be highly correlated with the later state $s_t$. For example, both may depend on earlier actions. Independence is not required. The argument works because, after conditioning on the history, the expected score of the newly sampled action is zero.

The statement concerns

\[\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t),\]

not the gradient of a joint marginal such as $\nabla_{\theta}\log p_{\theta}(a_t,s_t)$. The distribution of $s_t$ depends on previous policy decisions, so the latter contains additional terms.

From the proof to reward-to-go

Write

\[g_t=\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t).\]

Then

\[\begin{aligned} \nabla_{\theta}J &= \sum_t \mathbb{E} \left[ \left(\sum_k r_k\right)g_t \right] \\ &= \sum_t\sum_k\mathbb{E}[r_k g_t]. \end{aligned}\]

Every term with $k<t$ is zero. Consequently,

\[\begin{aligned} \nabla_{\theta}J &= \sum_t \mathbb{E} \left[ \left(\sum_{k=t}^{T-1}r_k\right)g_t \right] \\ &= \mathbb{E} \left[ \sum_t G_t \nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) \right], \end{aligned}\]

where

\[\boxed{ G_t=\sum_{k=t}^{T-1}r_k }\]

is the reward-to-go for the undiscounted objective. If the objective is

\[J_{\gamma}(\theta) = \mathbb{E} \left[ \sum_{k=0}^{T-1}\gamma^k r_k \right],\]

define the locally discounted reward-to-go

\[G_t^{(\gamma)} = \sum_{\ell=0}^{T-t-1}\gamma^{\ell}r_{t+\ell}.\]

The exact episodic gradient then contains the discount accumulated before time $t$:

\[\nabla_{\theta}J_{\gamma} = \mathbb{E} \left[ \sum_t \gamma^t G_t^{(\gamma)} \nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) \right].\]

Some presentations absorb this outer factor into the discounted state-visitation distribution, so it may not appear explicitly. The convention should be stated rather than silently mixed.

Reward-to-go is therefore not merely a heuristic. It is obtained by deleting terms whose expectation is exactly zero. This normally reduces variance because the score for $a_t$ is no longer multiplied by irrelevant random rewards from the past.

Baselines and Advantages

The same zero-score identity lets us subtract a baseline without changing the expected policy gradient. Let $b(H_t)$ be any quantity known before sampling $a_t$. Then

\[\mathbb{E} \left[ b(H_t)\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) \right] =0.\]

Thus,

\[\boxed{ \nabla_{\theta}J = \mathbb{E} \left[ \sum_t \left(G_t-b(H_t)\right) \nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) \right]. }\]

The baseline does not need to be constant. It can depend on the state or the full pre-action history. It must not depend on the current sampled action in a way that invalidates the zero-score argument.

If a learned baseline shares parameters with the policy, its value is normally detached in the actor loss. Otherwise, automatic differentiation adds a gradient through the baseline itself, which is not part of the policy-gradient identity above.

The most important baseline is the value function

\[V^{\pi}(s) = \mathbb{E}_{\pi}[G_t\mid s_t=s].\]

Define the action-value function

\[Q^{\pi}(s,a) = \mathbb{E}_{\pi}[G_t\mid s_t=s,a_t=a]\]

and the advantage

\[\boxed{ A^{\pi}(s,a)=Q^{\pi}(s,a)-V^{\pi}(s). }\]

The advantage asks a relative question: was this action better or worse than what the policy usually does in this state?

  • $A>0$: increase the action probability.
  • $A<0$: decrease the action probability.
  • $A=0$: this sample supplies no first-order preference.

The policy-gradient theorem can therefore be written in its familiar form [Sut99]:

\[\nabla_{\theta}J = \mathbb{E} \left[ A^{\pi}(s_t,a_t) \nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) \right].\]

The Critic and Generalized Advantage Estimation

The true value function is unknown. PPO normally learns a critic

\[V_{\psi}(s)\approx V^{\pi}(s).\]

A simple Monte Carlo advantage estimate is

\[\hat A_t=G_t-V_{\psi}(s_t).\]

This can have high variance, especially when rewards are delayed. Generalized Advantage Estimation (GAE) constructs a bias-variance trade-off from temporal-difference residuals [Sch15b]. Define

\[\delta_t = r_t+\gamma V_{\psi}(s_{t+1})-V_{\psi}(s_t).\]

If the critic is exact, then

\[\mathbb{E}[\delta_t\mid s_t,a_t] =A^{\pi}(s_t,a_t).\]

A $k$-step estimate is

\[\hat A_t^{(k)} = \sum_{\ell=0}^{k-1}\gamma^{\ell}\delta_{t+\ell}.\]

Expanding this sum makes the intermediate value terms cancel:

\[\hat A_t^{(k)} = -V(s_t) + \sum_{\ell=0}^{k-1}\gamma^{\ell}r_{t+\ell} + \gamma^k V(s_{t+k}).\]

GAE takes an exponentially weighted average of these multi-step estimates and simplifies to

\[\boxed{ \hat A_t^{\mathrm{GAE}(\gamma,\lambda)} = \sum_{\ell=0}^{\infty} (\gamma\lambda)^{\ell}\delta_{t+\ell}. }\]

In a finite episode it is computed backward:

\[\boxed{ \hat A_t = \delta_t+\gamma\lambda(1-d_t)\hat A_{t+1}, }\]

where $d_t$ indicates termination.

The extremes are intuitive:

\[\lambda=0 \quad\Longrightarrow\quad \hat A_t=\delta_t,\]

which relies strongly on the critic and usually has lower variance, while

\[\lambda=1 \quad\Longrightarrow\quad \hat A_t=G_t-V(s_t),\]

which approaches a Monte Carlo estimate and usually has higher variance.

Why PPO Introduces a Probability Ratio

Suppose trajectories were sampled from an old policy $\pi_{\mathrm{old}}$, but we are now optimizing a new policy $\pi_{\theta}$. At a fixed state,

\[\mathbb{E}_{a\sim\pi_{\theta}}[A_{\mathrm{old}}(s,a)] = \sum_a\pi_{\theta}(a\mid s)A_{\mathrm{old}}(s,a).\]

Insert the old policy:

\[\begin{aligned} \sum_a\pi_{\theta}(a\mid s)A_{\mathrm{old}}(s,a) &= \sum_a \pi_{\mathrm{old}}(a\mid s) \frac{\pi_{\theta}(a\mid s)} {\pi_{\mathrm{old}}(a\mid s)} A_{\mathrm{old}}(s,a). \end{aligned}\]

Define the importance ratio

\[\boxed{ \rho_t(\theta) = \frac{\pi_{\theta}(a_t\mid s_t)} {\pi_{\mathrm{old}}(a_t\mid s_t)}. }\]

This produces the surrogate objective

\[L^{\mathrm{CPI}}(\theta) = \mathbb{E}_{t\sim\pi_{\mathrm{old}}} [\rho_t(\theta)\hat A_t].\]

At the start of an update, when $\theta=\theta_{\mathrm{old}}$, we have $\rho_t=1$. Moreover,

\[\nabla_{\theta}\rho_t = \rho_t\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t),\]

so the gradient of the surrogate at the old policy is the ordinary advantage-weighted policy gradient.

The ratio corrects the action distribution at states sampled by the old policy. It does not fully correct the state-visitation distribution, which also changes when the policy changes. This is why the surrogate is most reliable for a local policy update. TRPO formalizes this idea with a KL-divergence trust region [Sch15a]. PPO replaces the constrained second-order update with a simpler clipped first-order objective [Sch17].

PPO’s Clipped Objective

PPO maximizes

\[\boxed{ L^{\mathrm{CLIP}}(\theta) = \mathbb{E}_t \left[ \min \left( \rho_t\hat A_t, \operatorname{clip}(\rho_t,1-\epsilon,1+\epsilon)\hat A_t \right) \right]. }\]

Optimizers usually minimize a loss, so implementations use

\[\mathcal{L}_{\mathrm{policy}} =-L^{\mathrm{CLIP}}.\]

The minimum makes the objective pessimistic: once a sampled action has moved far enough in the favorable direction, that sample stops rewarding an even larger move.

Positive advantage

When $\hat A_t>0$, increasing the action probability is favorable. The objective becomes

\[\ell_t = \hat A_t\min(\rho_t,1+\epsilon).\]

For $\rho_t>1+\epsilon$, the sample is saturated and supplies no further policy gradient.

Negative advantage

When $\hat A_t<0$, decreasing the action probability is favorable. Because multiplication by a negative value reverses the ordering,

\[\ell_t = \hat A_t\max(\rho_t,1-\epsilon).\]

For $\rho_t<1-\epsilon$, the sample is saturated and supplies no further policy gradient.

Clipping is asymmetric in an important way: it suppresses an excessively large favorable move, not an unfavorable move. If a positive-advantage action becomes less likely, PPO leaves the corrective gradient active.

Gradient of the PPO Loss

Ignoring the exact nondifferentiable clipping boundaries, define

\[m_t = \mathbf{1}[\hat A_t>0,\rho_t<1+\epsilon] + \mathbf{1}[\hat A_t<0,\rho_t>1-\epsilon].\]

Then

\[\boxed{ \nabla_{\theta}L^{\mathrm{CLIP}} = \mathbb{E}_t \left[ m_t\rho_t\hat A_t \nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) \right]. }\]

The minimized policy loss has the opposite sign:

\[\boxed{ \nabla_{\theta}\mathcal{L}_{\mathrm{policy}} =- \mathbb{E}_t \left[ m_t\rho_t\hat A_t \nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) \right]. }\]

This is still advantage-weighted log-likelihood. PPO only changes the coefficient by adding the old-to-new ratio and the clipping mask.

Gradient with respect to logits

Suppose the policy produces logits $z_j$ and probabilities

\[\pi_j=\frac{e^{z_j}}{\sum_k e^{z_k}}.\]

For the sampled action $a$,

\[\frac{\partial\log\pi(a\mid s)}{\partial z_j} = \mathbf{1}[j=a]-\pi_j.\]

In an active, unclipped region,

\[\frac{\partial\mathcal{L}_{\mathrm{policy},t}}{\partial z_j} = -\rho_t\hat A_t \left(\mathbf{1}[j=a_t]-\pi_j\right).\]

For positive advantage, gradient descent raises the selected action’s logit relative to the alternatives. For negative advantage, it lowers it.

The Full PPO Loss

A common minimized PPO objective is

\[\boxed{ \mathcal{L}_{\mathrm{PPO}} = \mathcal{L}_{\mathrm{policy}} +c_V\mathcal{L}_{\mathrm{value}} -c_H\mathcal{H}. }\]

The value loss is often

\[\mathcal{L}_{\mathrm{value}} = \frac{1}{2} \mathbb{E}_t \left[ \left(V_{\psi}(s_t)-\hat R_t\right)^2 \right],\]

where a common target is

\[\hat R_t=\hat A_t+V_{\mathrm{old}}(s_t).\]

The entropy term

\[\mathcal{H} = \mathbb{E}_s \left[ -\sum_a\pi_{\theta}(a\mid s) \log\pi_{\theta}(a\mid s) \right]\]

encourages exploration when it is subtracted from the minimized loss.

During the actor update, the old log-probabilities, returns, and advantages are treated as constants. The critic is trained through its separate value loss.

PPO for Language Models

For an autoregressive language model,

\[s_t=(q,o_{<t}),\qquad a_t=o_t.\]

The token-level PPO ratio is

\[\rho_t = \frac{ \pi_{\theta}(o_t\mid q,o_{<t}) }{ \pi_{\mathrm{old}}(o_t\mid q,o_{<t}) }.\]

It is evaluated stably in log-space:

\[\rho_t = \exp \left( \log\pi_{\theta}(o_t\mid q,o_{<t}) - \log\pi_{\mathrm{old}}(o_t\mid q,o_{<t}) \right).\]

A sequence-level ratio would multiply all token ratios. For long responses this product can become extremely large or small. Token-level PPO instead views every prefix as a state and every next token as an action.

Three models must be kept conceptually separate:

  1. The current policy $\pi_{\theta}$ is being optimized.
  2. The old policy $\pi_{\mathrm{old}}$ generated the current rollouts and appears in the PPO ratio.
  3. The reference policy $\pi_{\mathrm{ref}}$ regularizes behavior toward a stable language model, often the supervised fine-tuned model.

PPO also normally uses a fourth component, the critic $V_{\psi}$. The old policy and reference policy may initially contain identical weights, but they have different mathematical roles. The old policy is part of importance weighting; the reference policy is part of behavioral regularization.

For language-model RL, the task reward often arrives only after the complete answer. The critic attempts to predict the expected final outcome from each partial response. This can provide token-specific credit assignment, but it requires training and storing an additional value model. GRPO removes that component.

Group Relative Policy Optimization

GRPO was introduced as a critic-free variant of PPO in DeepSeekMath [Sha24]. For each prompt $q$, sample a group of $G$ responses from the old policy:

\[o_1,\ldots,o_G \sim \pi_{\mathrm{old}}(\cdot\mid q).\]

Score them with a reward function:

\[R_1,\ldots,R_G.\]

Compute prompt-local statistics

\[\mu_q = \frac{1}{G}\sum_{i=1}^G R_i,\] \[\sigma_q = \sqrt{ \frac{1}{G}\sum_{i=1}^G(R_i-\mu_q)^2 +\varepsilon_{\mathrm{std}} },\]

and define the group-relative advantage

\[\boxed{ \hat A_i = \frac{R_i-\mu_q}{\sigma_q}. }\]

Under outcome supervision, the original GRPO formulation assigns the same response-level advantage to every token in the response:

\[\hat A_{i,t}=\hat A_i.\]

The clipped policy term is then the same as in PPO:

\[\min \left( \rho_{i,t}\hat A_i, \operatorname{clip}(\rho_{i,t},1-\epsilon,1+\epsilon)\hat A_i \right),\]

where

\[\rho_{i,t} = \frac{ \pi_{\theta}(o_{i,t}\mid q,o_{i,<t}) }{ \pi_{\mathrm{old}}(o_{i,t}\mid q,o_{i,<t}) }.\]

A common response-normalized form of the objective is

\[\begin{aligned} J_{\mathrm{GRPO}}(\theta) = \mathbb{E} \Bigg[ \frac{1}{G} \sum_{i=1}^G \frac{1}{T_i} \sum_{t=1}^{T_i} \Bigg(& \min \left( \rho_{i,t}\hat A_i, \operatorname{clip}(\rho_{i,t},1-\epsilon,1+\epsilon)\hat A_i \right) \\ &-\beta\hat D_{\mathrm{KL},i,t} \Bigg) \Bigg]. \end{aligned}\]

The factor $1/T_i$ gives each response equal total weight rather than giving longer responses more weight merely because they contain more tokens.

GRPO’s KL Term

The original formulation uses the per-token estimator

\[\boxed{ \hat D_{\mathrm{KL}} = \frac{\pi_{\mathrm{ref}}(a\mid s)} {\pi_{\theta}(a\mid s)} - \log \frac{\pi_{\mathrm{ref}}(a\mid s)} {\pi_{\theta}(a\mid s)} -1. }\]

Let

\[x=\frac{\pi_{\mathrm{ref}}(a\mid s)} {\pi_{\theta}(a\mid s)}.\]

Then

\[\hat D_{\mathrm{KL}}=x-\log x-1\geq 0,\]

because $\log x\leq x-1$ for $x>0$.

If the action is sampled from the current policy, the expected value is the reverse KL divergence:

\[\begin{aligned} \mathbb{E}_{a\sim\pi_{\theta}}[\hat D_{\mathrm{KL}}] &= \sum_a\pi_{\theta}(a) \left[ \frac{\pi_{\mathrm{ref}}(a)}{\pi_{\theta}(a)} - \log\frac{\pi_{\mathrm{ref}}(a)}{\pi_{\theta}(a)} -1 \right] \\ &= D_{\mathrm{KL}} \left( \pi_{\theta}\,\|\,\pi_{\mathrm{ref}} \right). \end{aligned}\]

For a fixed sampled token, define

\[u = \log\pi_{\theta}(a\mid s) - \log\pi_{\mathrm{ref}}(a\mid s).\]

Then $x=e^{-u}$ and

\[\frac{\partial\hat D_{\mathrm{KL}}}{\partial u} =1-e^{-u}.\]

Therefore,

\[\boxed{ \nabla_{\theta}\hat D_{\mathrm{KL}} = \left( 1- \frac{\pi_{\mathrm{ref}}(a\mid s)} {\pi_{\theta}(a\mid s)} \right) \nabla_{\theta}\log\pi_{\theta}(a\mid s). }\]

The policy-gradient portion of GRPO is otherwise identical to PPO. Its active gradient is

\[\boxed{ \nabla_{\theta}J_{\mathrm{GRPO,policy}} = \mathbb{E} \left[ \frac{1}{G}\sum_i\frac{1}{T_i}\sum_t m_{i,t}\rho_{i,t}\hat A_i \nabla_{\theta} \log\pi_{\theta}(o_{i,t}\mid q,o_{i,<t}) \right]. }\]

The algorithmic difference is where $\hat A$ comes from:

\[\text{PPO: reward-to-go minus a learned value baseline,}\] \[\text{GRPO: response reward minus a same-prompt group baseline.}\]

What Group Normalization Changes

Group normalization has several consequences that are easy to miss.

The advantages are centered

Ignoring the numerical epsilon,

\[\sum_{i=1}^G\hat A_i=0.\]

GRPO therefore performs a relative update within each prompt. Some responses are pushed up and others are pushed down.

Positive affine reward transformations disappear

For $R_i’=aR_i+b$ with $a>0$,

\[\frac{R_i'-\mu'}{\sigma'} = \frac{R_i-\mu}{\sigma}.\]

This makes GRPO insensitive to prompt-specific reward offsets and positive scales. It also means absolute reward magnitude is discarded after normalization.

Equal rewards produce no reward gradient

If all responses receive the same reward, then every centered advantage is zero. The prompt supplies no reward-driven policy update. This can happen when all sampled answers are wrong, all are correct, or the reward function is too coarse.

With two samples, GRPO is close to ranking

With $G=2$, population standard deviation, and unequal rewards, the normalized advantages are $+1$ and $-1$. Only the ordering remains; the size of the reward gap disappears. A different standard-deviation convention changes the magnitude but not the direction.

The ordinary group mean contains the current sample

A textbook action-independent baseline should not depend on the sampled action whose score it multiplies. The group mean

\[\bar R=\frac{1}{G}\sum_jR_j\]

contains $R_i$ itself. For the unnormalized centered estimator

\[\hat g = \frac{1}{G} \sum_{i=1}^G (R_i-\bar R) \nabla\log\pi(o_i),\]

independent sampling gives

\[\mathbb{E}[\hat g] = \frac{G-1}{G} \mathbb{E} \left[ R\nabla\log\pi(o) \right].\]

Thus self-inclusion preserves the expected direction but scales the unnormalized estimator. A leave-one-out baseline

\[b_{-i} = \frac{1}{G-1}\sum_{j\neq i}R_j\]

is independent of response $i$ conditional on the prompt and removes this particular factor. Dividing by the random group standard deviation introduces additional stochastic rescaling, so normalized GRPO is best understood as a practical relative estimator rather than a literal unbiased copy of REINFORCE.

A Small Numerical GRPO Example

Suppose four responses to the same prompt receive

\[R=[1,0,0,0].\]

The group mean is

\[\mu=0.25.\]

Using population standard deviation,

\[\sigma = \sqrt{ \frac{(0.75)^2+3(-0.25)^2}{4} } \approx0.433.\]

The normalized advantages are

\[\hat A \approx [1.732,-0.577,-0.577,-0.577].\]

Every token in the successful response receives positive advantage, while tokens in the three unsuccessful responses receive negative advantage.

Now let $\epsilon=0.2$. For a token in the successful response with $\rho=1.10$,

\[\rho\hat A =1.10\times1.732 \approx1.905,\]

and the gradient is active. If $\rho=1.25$, PPO clipping limits the objective contribution to

\[1.2\times1.732 \approx2.078,\]

and that sample supplies no further favorable policy gradient beyond the clipping boundary.

PPO and GRPO Side by Side

Property PPO GRPO
Policy update Clipped probability-ratio objective Same clipped objective
Baseline Learned value function Same-prompt group statistics
Critic Normally required Not required
Advantage granularity Usually token- or time-specific through GAE Often one outcome advantage per response
Value loss Yes No
Rollouts Ordinary trajectories Multiple responses for each prompt
Main benefit Fine-grained learned credit assignment Lower memory and implementation complexity
Main weakness Critic cost and value-estimation error Coarse credit assignment and zero-signal groups

A compact summary is

\[\boxed{ \text{GRPO is a PPO-style update with a prompt-local Monte Carlo baseline.} }\]

It removes the critic. It does not remove the reward function, the rollout policy, PPO clipping, or necessarily the reference policy.

Minimal PyTorch-Style Implementation

The following code contains the mathematical core. Production implementations additionally need distributed rollout collection, padding masks, mixed precision, optimizer state management, and careful metric logging.

from __future__ import annotations

import torch
from torch import Tensor


def ppo_policy_loss(
    log_probs: Tensor,
    old_log_probs: Tensor,
    advantages: Tensor,
    mask: Tensor,
    *,
    clip_epsilon: float = 0.2,
) -> Tensor:
    """Clipped PPO policy loss over valid tokens."""
    old_log_probs = old_log_probs.detach()
    advantages = advantages.detach()
    mask = mask.to(log_probs.dtype)

    ratio = torch.exp(log_probs - old_log_probs)
    unclipped = ratio * advantages
    clipped = ratio.clamp(
        1.0 - clip_epsilon,
        1.0 + clip_epsilon,
    ) * advantages

    token_objective = torch.minimum(unclipped, clipped)
    valid_tokens = mask.sum().clamp_min(1.0)
    return -(token_objective * mask).sum() / valid_tokens


def grpo_loss(
    log_probs: Tensor,       # [batch, group, time]
    old_log_probs: Tensor,   # [batch, group, time]
    ref_log_probs: Tensor,   # [batch, group, time]
    rewards: Tensor,         # [batch, group]
    mask: Tensor,            # [batch, group, time]
    *,
    clip_epsilon: float = 0.2,
    kl_coefficient: float = 0.01,
    std_epsilon: float = 1e-6,
) -> Tensor:
    """Outcome-supervised, response-normalized GRPO loss."""
    old_log_probs = old_log_probs.detach()
    ref_log_probs = ref_log_probs.detach()
    rewards = rewards.detach()
    mask = mask.to(log_probs.dtype)

    group_mean = rewards.mean(dim=1, keepdim=True)
    group_variance = (rewards - group_mean).square().mean(
        dim=1,
        keepdim=True,
    )
    group_std = torch.sqrt(group_variance + std_epsilon)

    response_advantage = (
        (rewards - group_mean) / group_std
    ).detach()
    token_advantage = response_advantage.unsqueeze(-1)

    ratio = torch.exp(log_probs - old_log_probs)
    unclipped = ratio * token_advantage
    clipped = ratio.clamp(
        1.0 - clip_epsilon,
        1.0 + clip_epsilon,
    ) * token_advantage
    policy_token_loss = -torch.minimum(unclipped, clipped)

    log_ref_over_policy = ref_log_probs - log_probs
    kl_token_loss = (
        torch.exp(log_ref_over_policy)
        - log_ref_over_policy
        - 1.0
    )

    token_loss = policy_token_loss + kl_coefficient * kl_token_loss

    token_count = mask.sum(dim=-1).clamp_min(1.0)
    response_loss = (token_loss * mask).sum(dim=-1) / token_count
    return response_loss.mean()

The essential implementation rules are visible in the code:

  • Compute probability ratios as exponentiated log-probability differences.
  • Detach old log-probabilities and advantages.
  • Normalize GRPO rewards independently within each prompt group.
  • Apply the response mask before averaging.
  • Decide deliberately whether to weight responses equally or tokens equally.

Common Failure Modes

Confusing objective and loss signs

Papers usually write an objective to maximize. Deep-learning libraries usually minimize. The implementation therefore needs a leading minus sign on the clipped policy objective.

Dividing log-probabilities

The correct ratio is

\[\rho=\exp(\log\pi_{\theta}-\log\pi_{\mathrm{old}}),\]

not $\log\pi_{\theta}/\log\pi_{\mathrm{old}}$.

Backpropagating through rollout quantities

Old log-probabilities, sampled rewards, returns, and advantages are fixed data during the actor update. They should not retain a gradient path to trainable rollout models.

Confusing the old and reference policies

The old policy is the denominator of the PPO ratio. The reference policy is the target of KL regularization. Replacing one role with the other changes the algorithm.

Refreshing the old policy inside the optimization epoch

The old policy should remain fixed while the rollout batch is reused for several minibatch updates. Otherwise the denominator changes and the clipping interpretation no longer matches the collected data.

Treating clipping as a global KL guarantee

PPO clipping limits the incentive contributed by individual sampled actions. It does not mathematically guarantee that the complete policy remains inside a fixed KL ball. Learning rate, number of epochs, and correlated parameter changes still matter.

Normalizing GRPO rewards across unrelated prompts

The defining comparison is within the group generated for one prompt. Batch-wide normalization mixes prompt difficulty and changes the meaning of the advantage.

Ignoring zero-variance groups

When every response in a group receives the same reward, the normalized advantages vanish. Monitoring the fraction of zero-variance groups is therefore an important GRPO diagnostic.

A Unifying View of Post-Training

Many post-training methods can be understood as weighted log-likelihood updates:

\[\nabla_{\theta}J = \mathbb{E} \left[ c(q,o,t) \nabla_{\theta} \log\pi_{\theta}(o_t\mid q,o_{<t}) \right].\]

The coefficient $c$ defines the method.

For supervised fine-tuning,

\[c=1\]

on selected demonstration tokens.

For REINFORCE,

\[c=G_t.\]

For advantage actor-critic methods,

\[c=\hat A_t.\]

For active PPO samples,

\[c=m_t\rho_t\hat A_t.\]

For outcome-supervised GRPO,

\[c=m_{i,t}\rho_{i,t} \frac{R_i-\mu_q}{\sigma_q}.\]

This perspective makes the algorithms less mysterious. The neural-network backpropagation is ordinary log-likelihood backpropagation. Most of the conceptual work lies in constructing a useful, low-variance, stable coefficient for every sampled token.

Conclusion

The path from policy gradients to PPO and GRPO consists of a small number of reusable ideas.

First, the log-derivative trick turns the gradient of expected reward into reward-weighted log-probability gradients. Second, causality removes past rewards: for $k<t$,

\[\mathbb{E} \left[ r_k\nabla\log\pi(a_t\mid s_t) \right]=0,\]

because the expected score of the newly sampled action is zero after conditioning on the history. This produces reward-to-go.

Third, the same score identity permits action-independent baselines, leading to advantages. PPO estimates those advantages with a critic, often through GAE, and stabilizes data reuse with an old-policy ratio and clipping. GRPO keeps the clipped PPO machinery but replaces the learned value baseline with a relative baseline constructed from several responses to the same prompt.

At the start of either update, when the current and old policies coincide, the central instruction is simple:

\[\boxed{ \text{Increase the probability of positive-advantage tokens and decrease the probability of negative-advantage tokens.} }\]

PPO and GRPO differ mainly in how they decide which sampled tokens deserve those positive or negative weights, and in how they prevent repeated optimization from moving the policy too aggressively.

References

[Wil92] R. J. Williams. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning, 1992.

[Sut99] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation. NeurIPS, 1999.

[Sch15a] J. Schulman, S. Levine, P. Moritz, M. I. Jordan, and P. Abbeel. Trust Region Policy Optimization. 2015.

[Sch15b] J. Schulman, P. Moritz, S. Levine, M. I. Jordan, and P. Abbeel. High-Dimensional Continuous Control Using Generalized Advantage Estimation. 2015.

[Sch17] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal Policy Optimization Algorithms. 2017.

[Sha24] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. 2024.