Human values are diverse, whereas LLMs are not.

Motivation

First recall Bradley-Terry Preference Model:

\begin{equation} p\qty(y^{\qty(a)} \succ y^{\qty(b)} \mid x) = \sigma \qty[ \qty(x,y^{(a)}) - r\qty(x, y^{(b)}) ] \end{equation}

in the sense that, there is no clear model of a user beyond the mere inputs.

User-Specific Reward Modeling

Let \(z\) be some latent about the user. So we can write:

\begin{equation} p\qty(y^{(a)} \succ y^{(b)} \mid x, z) = \sigma \qty[r\qty(x,y^{(a)} | z)- r\qty(x,y^{(b)} | z)] \end{equation}

Key Sequence

Notation

New Concepts

Important Results / Claims

Questions

Interesting Factoids

1. user preferences: do what the user wants in a way they like
2. safety: preventing models from causing harm during training and deployment
3. creator preferences: when the model developer wants the model to believe

Modern AI pipelines uses a joint grader infrastructure:

• Helpfulness
• Honestly
• Safety
• Sycophancy
• Format and length
• Code correctness
• RLHF

and they are jointly summed and then produced as the output.

Compound Systems

1. classifiers: small, fast models that score inputs and outputs for policy violation
2. monitoring: sampled human review, anomalously detection

Human feedbacks are often not correlated among each other nor with human judgement.

Adversarial testing

1. internal red teams: domain experts probe pre-release checkpoints, slow, expensive…
2. bug bounties: check a deployed system under standard rules of engagement
3. automated red teaming: attacker model searches for prompts that make a target fail

A/B testing, try to see how many people use the app

constituents

• policy \(y \sim \pi_{\theta}\qty(\cdot|x)\)
• reward \(R_{\phi}\qty(x,y) \to \mathbb{R}\)

requirements

1. give a prompt, we sample two rollouts \(y_1, y_2 \sim \pi_{\theta}\qty(\cdot | x)\)
2. labeler picks the better one \(y_{2} \succ y_{1}\)
3. comparisons go and train a reward model \(r_{\phi}\qty(x,y)\)

additional information

training reward models

Formally, we can model the preferability of the stitutaion with bradly-terry:

\begin{equation} p\qty(y_{w} \succ y_{l} | x) = \sigma \qty(r_{\phi}\qty(x, y_{w}) - r_{\phi}\qty(x, y_{l})) \end{equation}

You can then perform MLE on this with a Bernoulli distribution to train your \(r_{\phi}\).