no single person can decide aggregation

Stanford UG Y1, Aut

• Decision Making Index
• Computer Systems Index
• CS Probability Index
• Speech Processing Index

Stanford UG Y1, Win

• ODEs Index
• OS Index
• POMDPs Index
• Language Information Index

Stanford UG Y1, Spr

• Optimization Index
• Rust Systems Index
• Signal Processing Index
• NLP Index
• Democracy Index

Stanford UG Y2, Aut

• PL Index
• Theory of Computing Index

Stanford UG Y2, Win

• Complexity Theory Index
• ML Math Index
• Validation Index

Stanford UG Y2, Spr

• Complexity Theory Index
• Compilers Index
• Moral Philosophy Index

Stanford UG Y3, Aut

• Machine Learning Index
• Algorithms Index

Stanford UG Y3, Win

• Medical Ethics Index
• China Economy Index
• Convex Optimization Index
• Gender and Sexuality in East Asia

Stanford UG Y3, Spr

• Geology Index

Stanford GR Y1, Aut

• MLHF Index
• Agents Index

Stanford Talks

Contacts

Talk Contacts

Zero-Shot Learning

GPT-2 is able to do many tasks with not examples + no gradient updates.

Instruction Fine-Tuning

Language models, by default, are not aligned with user intent.

1. collect paired examples of instruction + output across many tasks
2. then, evaluate on unseen tasks

~3 million examples << n billion examples

dataset: MMLU

You can generate an Instruction Fine-Tuning dataset by asking a larger model for it (see Alpaca).

Pros + Cons

• simple and straightforward + generalize to unseen tasks
• but, its EXPENSIVE to collect ground truth data
  • ground truths maybe wrong
  • creative tasks may not have a correct answer
  • LMs penalizes all token-level mistakes equally, but some mistakes are worse than others
  • humans may generate suboptimal answers

Human Preference Modeling

Imagine if we have some input \(x\), and two output trajectories, \(y_{1}\) and \(y_{2}\).

MLHF!!!

Optimizing system used by humans, from data on what they chose

New Concepts

• LMArena

Important Results / Claims

• Luce’s Axiom
• Impossibility of Aggregation aka Arrow’s Theorem

Rough Fields / Mapping

Book: mlhp.stanford.edu

Foundations

“where does the data come from”

• Psycometrics: how to measure people
• Discrete choice: how to do the math

Learning

“how do we model and fit”

• Reinforcement learning: how to use what you’ve measured
• Decision theory / inverse RL: POMDP stuff, since RL model need to actually map to actions

action

“what do we do with the fit”

> nit

Slide numbers :)

> How having a probability related to generation? / Why is LM discriminative.

Re the student question about why a language model isn’t descriminative. It’s sorta true that LMs are a “classifier”, as are many descriminative models, but notice that there’s no “correct” classification.

Instead, you are normatively “supposed” to sample the distribution instead of simply outputting the “correct” next token because no such token exist. In this sense, its generative.