Floating Point

4 bytes

\begin{equation} (-1)^{B} + e^{E-127} \times \qty(1 + \sum_{i=1}^{23} b_{23-i}2^{-i}) \end{equation}

usually \(E\) is a 8 bytes, and 23 digits of \(b\).

With more \(E\), we will have more range, with more \(b\), we will have more precision.

Mixed Precision Training

1. copy the model in FP32
2. Run forward pass in FP16
3. Scale loss to be large enough to not be rounded away
4. Compute gradients in FP16
5. Convert the gradients onto FP32
6. Scale the gradients down
7. apply to the model

BFloat16

To not need to scale, we can use a scheme that has less precision but the same amount of dynamic range (i.e. allocate the same \(E\), chop off \(b\)) —no need to scale, just have more dynamic range.

Key Information

• Title: Fine-Grained Language Model Detoxification with Dense, Token-Level Rewards
• Team Member (in 224n): Houjun Liu <[email protected]>
• External Collaborators: Amelia Hardy <[email protected]>, Bernard Lange <[email protected]>
• Custom Project
• Mentor: we have no particular mentor within 224n
• Sharing Project: this project is shared with AA222, and is a part of a research project PI’d by Mykel Kochenderfer <[email protected]>, of which Houjun is taking a leading role

Research Paper Summary

Background

Reinforcement Learning with Human Feedback (RLHF) has demonstrated superb effect for improving performance of a language model (LM) via human preference judgments of LM output desirability–reducing incidences of toxic or false generation trajectories ((Ziegler et al. 2020)). Naive application of RLHF directly has shown success in reducing the toxicity in language model outputs, yet its effects could sometimes be inconsistent without further in-context guidance of the resulting model ((Ouyang et al. 2022)).

• machine learning
  • types of ML
  • history of ML
  • Key Ideas of ML
  • house price prediction: linear regression
  • ImageNet
    • ImageNet Large Scale Visual Recognition Challenge (ILSVRC)
  • Language Model

Notation

New Concepts

• supervised learning
  • training set
  • cost function
    • least-squares error
• machine learning evaluation
  • root-mean-square error (this is least-squares error with normalization, so “units work well” for test benchmarks)
• linear regression
• gradient descent
  • stochastic gradient descent
• normal equation

Important Results / Claims

• gradient descent for least-squares error
• when does gradient descent provably work?

Questions

• Why can’t we use root-mean-square error for the training objective? It seems like its just more normalization…

Key Sequence

Notation

New Concepts

• Optimal Stopping Problem
• reinforcement learning
  • model-based reinforcement learning
  • model-free reinforcement learning

Important Results / Claims

Questions

Interesting Factoids