AISafety2025 Bansal: Safety Constrained Sets

Detect tokens (latents?) which trigger potential paths into unsafe behavior, and then preempt them early by steering.

Intuition:

• \(O\): \(\leq\) (function in the symbol is
• \(\theta\): \(=\)
• \(\Omega\): \(\geq\) (function in the symbol is a lower bound)

Definition Intuition:

We say \(f\qty(n) = O\qty(g\qty(n))\) such that “when \(n\) gets big enough, \(f\qty(n)\) is bounded by at most a constant multiple of \(g\qty(n)\).

Definitions:

• \(f(n) = O(g(n)) \Leftrightarrow \exists c, n_{0} > 0: \forall n > n_0, f(n) \leq c (g(n))\)
• \(f(n) = \Omega(g(n)) \implies \exists n_{0}: \forall n > n_0, f(n) \geq c (g(n))\)
• \(f(n) = \theta(g(n)) \implies \exists n_{0}: \forall n > n_0, f(n) \geq 1 (g(n)), f(n) \leq c (g(n))\)

Little ones:

a cost function \(J\) tells us how good our training is. For instance, least-squares error

additional information

It’s hard to make globally optimal solution, so therefore we instead make local progress.

constituents

• parameters \(\theta\)
• step size \(\alpha\)
• cost function \(J\) (and its derivative \(J’\))

requirements

let \(\theta^{(0)} = 0\) (or a random point), and then:

\begin{equation} \theta^{(t+1)} = \theta^{(t)} - \alpha J’\qty (\theta^{(t)}) \end{equation}

“update the weight by taking a step in the opposite direction of the gradient by weight”. We stop, btw, when its “good enough” because the training data noise is so much that like a little bit non-convergent optimization its fine.

Lecture

• SU-MED103 SEP252025