Gen AI Safety Model and Datasets
Last edited: September 9, 2026Safety Model Information
I/O Guardrails
- input guardrail: check query before processing to see if its safe to answer
- output guardrail (“Shield Reviewer”): scans the generated content chunk-by-chunk to ensure the output remains compliant
Launch Guardrails
“Safety Auto raters” ensure new features don’t compromise safety without red teaming:
How to classify?
high priority launch: policy changes, new features, business interest
low priority launch: auto-rated
User Feedback
Half a million user feedback point daily. How to filter? For both safety and quality outputs.
Inference Fabrics Index
Last edited: September 9, 2026Lectures
non-dictatorship
Last edited: September 9, 2026no single person can decide aggregation
SU-CS329H SEP232026
Last edited: September 9, 2026MLHF!!!
Optimizing system used by humans, from data on what they chose
New Concepts
Important Results / Claims
Rough Fields / Mapping
Book: mlhp.stanford.edu
Foundations
“where does the data come from”
- Psycometrics: how to measure people
- Discrete choice: how to do the math
Learning
“how do we model and fit”
- Reinforcement learning: how to use what you’ve measured
- Decision theory / inverse RL: POMDP stuff, since RL model need to actually map to actions
action
“what do we do with the fit”
SU-CS329Z SEP282026
Last edited: September 9, 2026> nit
Slide numbers :)
> How having a probability related to generation? / Why is LM discriminative.
Re the student question about why a language model isn’t descriminative. It’s sorta true that LMs are a “classifier”, as are many descriminative models, but notice that there’s no “correct” classification.
Instead, you are normatively “supposed” to sample the distribution instead of simply outputting the “correct” next token because no such token exist. In this sense, its generative.
