_index.org

Gen AI Safety Model and Datasets

Last edited: September 9, 2026

Safety Model Information

I/O Guardrails

  • input guardrail: check query before processing to see if its safe to answer
  • output guardrail (“Shield Reviewer”): scans the generated content chunk-by-chunk to ensure the output remains compliant

Launch Guardrails

“Safety Auto raters” ensure new features don’t compromise safety without red teaming:

How to classify?

high priority launch: policy changes, new features, business interest
low priority launch: auto-rated

User Feedback

Half a million user feedback point daily. How to filter? For both safety and quality outputs.

Inference Fabrics Index

Last edited: September 9, 2026

Lectures

non-dictatorship

Last edited: September 9, 2026

no single person can decide aggregation

SU-CS329H SEP232026

Last edited: September 9, 2026

MLHF!!!

Optimizing system used by humans, from data on what they chose

New Concepts

Important Results / Claims

Rough Fields / Mapping

Book: mlhp.stanford.edu

Foundations

“where does the data come from”

  • Psycometrics: how to measure people
  • Discrete choice: how to do the math

Learning

“how do we model and fit”

  • Reinforcement learning: how to use what you’ve measured
  • Decision theory / inverse RL: POMDP stuff, since RL model need to actually map to actions

action

“what do we do with the fit”

SU-CS329Z SEP282026

Last edited: September 9, 2026

> nit

Slide numbers :)

> How having a probability related to generation? / Why is LM discriminative.

Re the student question about why a language model isn’t descriminative. It’s sorta true that LMs are a “classifier”, as are many descriminative models, but notice that there’s no “correct” classification.

Instead, you are normatively “supposed” to sample the distribution instead of simply outputting the “correct” next token because no such token exist. In this sense, its generative.