1. user preferences: do what the user wants in a way they like
2. safety: preventing models from causing harm during training and deployment
3. creator preferences: when the model developer wants the model to believe

Modern AI pipelines uses a joint grader infrastructure:

• Helpfulness
• Honestly
• Safety
• Sycophancy
• Format and length
• Code correctness
• RLHF

and they are jointly summed and then produced as the output.

Compound Systems

1. classifiers: small, fast models that score inputs and outputs for policy violation
2. monitoring: sampled human review, anomalously detection

Human feedbacks are often not correlated among each other nor with human judgement.

Adversarial testing

1. internal red teams: domain experts probe pre-release checkpoints, slow, expensive…
2. bug bounties: check a deployed system under standard rules of engagement
3. automated red teaming: attacker model searches for prompts that make a target fail

A/B testing, try to see how many people use the app

constituents

• policy \(y \sim \pi_{\theta}\qty(\cdot|x)\)
• reward \(R_{\phi}\qty(x,y) \to \mathbb{R}\)

requirements

1. give a prompt, we sample two rollouts \(y_1, y_2 \sim \pi_{\theta}\qty(\cdot | x)\)
2. labeler picks the better one \(y_{2} \succ y_{1}\)
3. comparisons go and train a reward model \(r_{\phi}\qty(x,y)\)

additional information

training reward models

Formally, we can model the preferability of the stitutaion with bradly-terry:

\begin{equation} p\qty(y_{w} \succ y_{l} | x) = \sigma \qty(r_{\phi}\qty(x, y_{w}) - r_{\phi}\qty(x, y_{l})) \end{equation}

You can then perform MLE on this with a Bernoulli distribution to train your \(r_{\phi}\).

Zero-Shot Learning

GPT-2 is able to do many tasks with not examples + no gradient updates.

Instruction Fine-Tuning

Language models, by default, are not aligned with user intent.

1. collect paired examples of instruction + output across many tasks
2. then, evaluate on unseen tasks

~3 million examples << n billion examples

dataset: MMLU

You can generate an Instruction Fine-Tuning dataset by asking a larger model for it (see Alpaca).

Pros + Cons

• simple and straightforward + generalize to unseen tasks
• but, its EXPENSIVE to collect ground truth data
  • ground truths maybe wrong
  • creative tasks may not have a correct answer
  • LMs penalizes all token-level mistakes equally, but some mistakes are worse than others
  • humans may generate suboptimal answers

Human Preference Modeling

Imagine if we have some input \(x\), and two output trajectories, \(y_{1}\) and \(y_{2}\).

Key Sequence

Notation

New Concepts

• Goals of Alignment
• RLHF
• DPO

Important Results / Claims

why not prompting

Even if its fast, no training needed, but its still not good for complex / nuanced behaviors. Models stop following long prompts, and rewards models are very very specific.

“Articulability”

Questions

Interesting Factoids