constituents

• policy \(y \sim \pi_{\theta}\qty(\cdot|x)\)
• reward \(R_{\phi}\qty(x,y) \to \mathbb{R}\)

requirements

1. give a prompt, we sample two rollouts \(y_1, y_2 \sim \pi_{\theta}\qty(\cdot | x)\)
2. labeler picks the better one \(y_{2} \succ y_{1}\)
3. comparisons go and train a reward model \(r_{\phi}\qty(x,y)\)

additional information

training reward models

Formally, we can model the preferability of the stitutaion with bradly-terry:

\begin{equation} p\qty(y_{w} \succ y_{l} | x) = \sigma \qty(r_{\phi}\qty(x, y_{w}) - r_{\phi}\qty(x, y_{l})) \end{equation}

You can then perform MLE on this with a Bernoulli distribution to train your \(r_{\phi}\).

reinforcement learning

Now you have a reward of utilities, notice that its now just a utility score of a sequence independent of whether a pair is preferable, so you can just do it.

You also often train on a much larger body of prompts etc. Specifically, you want:

\begin{equation} \max_{\theta} \mathbb{E}_{x,y}\qty [r_{\phi}\qty(x,y)] - \beta \mathbb{E}_{x} \qty [D_{\text{KL}}\qty( \pi_{\theta}\qty(\cdot|x) \mid \pi_{\text{ref}} \qty(\cdot | x))] \end{equation}