Human values are diverse, whereas LLMs are not.

Motivation

First recall Bradley-Terry Preference Model:

\begin{equation} p\qty(y^{\qty(a)} \succ y^{\qty(b)} \mid x) = \sigma \qty[ \qty(x,y^{(a)}) - r\qty(x, y^{(b)}) ] \end{equation}

in the sense that, there is no clear model of a user beyond the mere inputs.

User-Specific Reward Modeling

Let \(z\) be some latent about the user. So we can write:

\begin{equation} p\qty(y^{(a)} \succ y^{(b)} \mid x, z) = \sigma \qty[r\qty(x,y^{(a)} | z)- r\qty(x,y^{(b)} | z)] \end{equation}

Under a Bradly-Terry model, we may have \(p\qty(y^{\qty(a)} \succ y^{(b)} \mid x) = 0.5\), but it could be that conditioned on some \(z\) that the shift in terms of likelihood is very lopsided. For instance:

Question: how do we best represent your choice of \(z\)?

Inducing user information \(z\)

Task: how do we discover a good choice of \(z\) to induce specific rewards above?

• in-context learning through examples
• variation preference learning
• textual representation of user

in-context user information learning

Suppose you have some past information about the users, like:

\begin{equation} \qty(x, y_{l}, y_{w} \qty{x^{(i)}, y^{(i)}_{l}, y^{(i)}_{w}}) \in D \end{equation}

As a very simple way to do this: we can just stick all of \(D\) as a part of \(x\), and your model becomes:

\begin{equation} r_{\phi}\qty(x,y, D) \end{equation}

but then…. this will be redundant, inefficient, and really really long.

variational preference learning

We can learn a user embedding by assuming that there exists some latent stricture of the preferences. So we train a user encoder:

\begin{equation} z \sim q_{\psi}\qty( \cdot | D) \end{equation}

Then we use \(r_{\phi}\qty(x,y,z)\). We jointly train \(\psi\) and \(\phi\) together. We learn evidence lower bound:

\begin{equation} -\mathbb{E}_{D} \qty [\mathbb{E}_{z \sim q} \qty{\log p_{\phi} \qty(y_{w} \succ y_{l} | x, z) - \text{KL}\qty(q_{\phi}\qty(z \mid D \mid \mid p\qty(z)))}] \end{equation}

this maximizes \(p_{\phi}\qty(y_{w} \succ y_{l})\) with the help of a condition on \(z\).

textual user summaries

Get a summary language model \(\pi_{\psi}\), and then we:

1. generate a summary: \(z \sim \pi_{\psi}\qty(\cdot | D)\)
2. use the reward model’s loss to train the summarizer: the current reward model gets a loss \(\ell\qty(\phi)\), we’d like to maximize \(-L\qty(\psi) - \beta \text{KL}\qty(\pi_{\psi} \mid \pi_{\text{ref}})\)
3. train the reward model: minimize \(L = -\log p_{\phi}\qty(y_{w} \succ y_{l} | x, z)\)