For weighted sum method for instance, we need to figure a \(w\) such that:

\begin{equation} f = w^{\top}\mqty[f_1 \\ \dots\\f_{N}] \end{equation}

where weight \(w \in \triangle_{N}\).

To do this, we essentially infer the weighting scheme by asking “do you like system \(a\) or system \(b\)”.

1. first, we collect a series of design variables \((a_1, a_2, a_3 …)\) and \((b_1, b_2, b_3…)\) and we ask “which one do you like better”
2. say our user WLOG chose \(b\) over \(a\)
3. so we want to design a \(w\) such that \(w^{\top} a < w^{\top} b\)
4. meaning, we solve for a \(w\) such that…

\begin{align} \min_{w}&\ \sum_{i=1}^{n} (a_{i}-b_{i})w^{\top} \\ \text{such that}&\ \bold{1}^{\top} w = 1 \\ &\ w \geq 0 \end{align}

• autonomy
• informed consent
• beneficence
• non-maleficence
• justice

Thinking about advances in the capabilities of RL: Knowledge Discovery -> Reasoning (programming assistance) ->(ongoing)-> Robotics

Insight: as time goes on, the “risk-criticality” of our applications increase; yet, as risk critical scenarios increase, its harder to get data.

Reliable Feedback Loop

General desirable structure…

Verify (claims and requirements) => Safeguard (safe continuous deployment) => Generalize (via compositional generalization—incrementing adding behavior without loosing behavior) => Verify => …

Deal with Stochasticity

An RL algorithm is explicable, if, WHP, running on the same MDP with fixed randomness results in the same outcomes.

Key Sequence

Notation

New Concepts

• function convexity conditions
  • 1st order condition
  • 2nd order condition
• sublevel set
• epigraph
• Jensen’s Inequality

Important Results / Claims

• operations that preserve fuction convexity
• composition with scalar functions
• general composition rule that preserve convexity
• constructive convexity verification

Questions

Interesting Factoids

Challenge of moral philosophy: a system which resolves morality and ethics together.

tools

• definitions
• appearance vs. reality (descriptive vs. true values)
• reflective equilibrium

morality

morality is paradimatically a set of rules/expectations (concerning character/motives/emotions) for right/wrong behavior.

A code of conduct that people “must” follow to…

• regulate / guide interpersonal interactions
• rules that concern…
  • harm / benefit
  • justice / fairness
  • loyalty / obedience
  • sanctity / purity

Key question of morality: what do we owe to each other? what do I owe other people?