Modern AI pipelines uses a joint grader infrastructure:
- Helpfulness
- Honestly
- Safety
- Sycophancy
- Format and length
- Code correctness
- RLHF
and they are jointly summed and then produced as the output.
Compound Systems
- classifiers: small, fast models that score inputs and outputs for policy violation
- monitoring: sampled human review, anomalously detection
Human feedbacks are often not correlated among each other nor with human judgement.
Adversarial testing
- internal red teams: domain experts probe pre-release checkpoints, slow, expensive…
- bug bounties: check a deployed system under standard rules of engagement
- automated red teaming: attacker model searches for prompts that make a target fail
A/B testing, try to see how many people use the app