1. user preferences: do what the user wants in a way they like
2. safety: preventing models from causing harm during training and deployment
3. creator preferences: when the model developer wants the model to believe