Key Information

• Title: Fine-Grained Language Model Detoxification with Dense, Token-Level Rewards
• Team Member (in 224n): Houjun Liu <[email protected]>
• External Collaborators: Amelia Hardy <[email protected]>, Bernard Lange <[email protected]>
• Custom Project
• Mentor: we have no particular mentor within 224n
• Sharing Project: this project is shared with AA222, and is a part of a research project PI’d by Mykel Kochenderfer <[email protected]>, of which Houjun is taking a leading role

Research Paper Summary

Background

Reinforcement Learning with Human Feedback (RLHF) has demonstrated superb effect for improving performance of a language model (LM) via human preference judgments of LM output desirability–reducing incidences of toxic or false generation trajectories ((Ziegler et al. 2020)). Naive application of RLHF directly has shown success in reducing the toxicity in language model outputs, yet its effects could sometimes be inconsistent without further in-context guidance of the resulting model ((Ouyang et al. 2022)).

• machine learning
  • types of ML
  • history of ML
  • Key Ideas of ML
  • house price prediction: linear regression
  • ImageNet
    • ImageNet Large Scale Visual Recognition Challenge (ILSVRC)
  • Language Model

Notation

New Concepts

• supervised learning
  • training set
  • cost function
    • least-squares error
• machine learning evaluation
  • root-mean-square error (this is least-squares error with normalization, so “units work well” for test benchmarks)
• linear regression
• gradient descent
  • stochastic gradient descent
• normal equation

Important Results / Claims

• gradient descent for least-squares error
• when does gradient descent provably work?

Questions

• Why can’t we use root-mean-square error for the training objective? It seems like its just more normalization…

Key Sequence

Notation

New Concepts

• Optimal Stopping Problem
• reinforcement learning
  • model-based reinforcement learning
  • model-free reinforcement learning

Important Results / Claims

Questions

Interesting Factoids

Key Sequence

Notation

New Concepts

• POMDP
  • belief
  • observation model
    • error model
    • discrete state filter

Important Results / Claims

Questions

Interesting Factoids