[redirect] normal equation
Last edited: September 9, 2025See Normal Equation
for small equations of Linear Regression, we can solve it using normal equation method.
Consider \(d\) dimensional feature and \(n\) samples of data. Remember, including the dummy feature, we have a matrix: \(X \in \mathbb{R}^{n \times \qty(d+1)}\) and a target \(Y \in \mathbb{R}^{n}\).
Notice:
\begin{equation} J\qty(\theta) = \frac{1}{2} \sum_{i=1}^{n} \qty(h_{\theta} \qty(x^{(i)}) - y^{(i)})^{2} \end{equation}
and \(h = X \theta\), we we can write:
\begin{equation} J(\theta) = \frac{1}{2} \qty(X \theta - y)^{T} \qty(X \theta - y) \end{equation}
AI Safety Annual Meeting 2025
Last edited: September 9, 2025AISafety2025 Bansal: Safety Constrained Sets
Detect tokens (latents?) which trigger potential paths into unsafe behavior, and then preempt them early by steering.
big-o
Last edited: September 9, 2025Intuition:
- \(O\): \(\leq\) (function in the symbol is
- \(\theta\): \(=\)
- \(\Omega\): \(\geq\) (function in the symbol is a lower bound)
Definition Intuition:
We say \(f\qty(n) = O\qty(g\qty(n))\) such that “when \(n\) gets big enough, \(f\qty(n)\) is bounded by at most a constant multiple of \(g\qty(n)\).
Definitions:
- \(f(n) = O(g(n)) \Leftrightarrow \exists c, n_{0} > 0: \forall n > n_0, f(n) \leq c (g(n))\)
- \(f(n) = \Omega(g(n)) \implies \exists n_{0}: \forall n > n_0, f(n) \geq c (g(n))\)
- \(f(n) = \theta(g(n)) \implies \exists n_{0}: \forall n > n_0, f(n) \geq 1 (g(n)), f(n) \leq c (g(n))\)
Little ones:
cost function
Last edited: September 9, 2025a cost function \(J\) tells us how good our training is. For instance, least-squares error
additional information
gradient descent
Last edited: September 9, 2025It’s hard to make globally optimal solution, so therefore we instead make local progress.
constituents
- parameters \(\theta\)
- step size \(\alpha\)
- cost function \(J\) (and its derivative \(J’\))
requirements
let \(\theta^{(0)} = 0\) (or a random point), and then:
\begin{equation} \theta^{(t+1)} = \theta^{(t)} - \alpha J’\qty (\theta^{(t)}) \end{equation}
“update the weight by taking a step in the opposite direction of the gradient by weight”. We stop, btw, when its “good enough” because the training data noise is so much that like a little bit non-convergent optimization its fine.
