Group-relative policy optimisation

Consider the problem of having to optimise the parameters $\theta$ for a policy $\pi_{\theta}(a|s)$. We know from policy gradient (Williams, 1992) that x.

$$$$

We know that using standard policy gradient methods are unbiased, but have high variance. Here we like to use variance reduction methods which can be encapsulated using the generalised advantage estimation framework. However we have seen success only with learned baselines such as value function or Q-functions. What if we could skip that all together and still have low variance?

Enter GRPO in which we use rollouts that form a group and estimate the advantage of each rollout relative to the group average.

$$A(s, a_j) = \frac{r(s, a_j) - \mu}{\sigma}$$

Where $\mu = \frac{1}{n} \sum_{i=1}^n r(s, a_i)$, $\sigma = \sqrt{\frac{1}{n} \sum_{i=1}^n (r(s, a_i) - \mu)^2}$. Notice that the the advantage is undefined for $\sigma = 0$, which happens when either all $r(s, a_i)$ coincide or if our group size $n$ is too small ($n=1$). Note that the normalisation function is arbitrary but works well in practice.