Two problems: scarcity and alignment

scarcity

=> you can’t RL humans freely

• …so the data is selected given priors of how the humans showed up
• we thus need to make assumptions to generalize

Why you need assumptions since you have scarcity

We’d love to infer what X will do given Y circumstance, but unless you have a thing that’s positively X (i.e. a box that’s literally X you can replay), you have to use your experience of (X’, Y’) to make an inductive prior for what X will do.

Exceptions

• social media: you can RL on humans a bit, its however narrow, with limited actions
• medical trials, etc.

alignment

=> we don’t understand humans

• how do you personalize something for people?
• how do you delegate to agents?