model function, nearest neighbor method, data interpolation, overfitting and underfiting, regularization

noise vs. features

If you know a property of the underlying distribution, noise and the important underlying feature could look very similar (think a sinusoid vs. just Gaussian noise).

learning

learning is the practice of training a system to do something….

knowledge based system

A John McCarthy-esque learning system using knowledge: 1) knowledge base, with explicit facts with experts and 2) an inference engine, a way to reason and generate new facts.

Key Sequence

Review

• data interpolation
• regularization

Notation

New Concepts

• Train-Test Split
• Robustness
  • well-posedness
• Numerical Stability
  • numerically stable vector norms
  • numerically stable quadratic formula
• polynomial interpolation
  • Vandermonde matrix + monomial basis
  • Lagrange Basis
  • Newton Basis

Important Results / Claims

• near-singular matrix problem

Questions

Interesting Factoids

• “optimization is a wrong way to train a neural network” (because you can’t hope to sample a trajectory in \(\mathbb{R}^{n}\) at large \(n\))

Key Sequence

Notation

New Concepts

• using domain knowledge: skinning
  • displacement map
• Linear Systems
  • Systems of Linear Equations
  • problems with zero
  • square matrix
  • Diagonal Matrix
  • upper-triangular matricies
• Elimination Matricies
  • pivoting
  • LU-Factorization
  • permutation matricies
• sparsity

Important Results / Claims

• linear systems with diagonal matricies

Questions

Interesting Factoids

Random properties of eigenthings…

Hermitian Matrix

A matrix such that: \(A^{*^{T}} = A\)

\begin{equation} v^{*^{T}} A = \lambda^{*} v^{*^{T}} \end{equation}

Meaning; \(Av = \lambda v \implies \qty(v^{*^{T}} A v = v^{*^{T}} \lambda v) \implies \lambda^{*} = \lambda\)

Symmetric matrices have this property.

vector deformation

Suppose \(v\) are the eigenvectors of \(A\), some vector \(c = \sum_{k}^{} a_{k} v_{k}\) (as long as they are from different eigenvalues), and so, applying \(A\):

\begin{equation} Ac = \sum_{k}^{} a_{k} A v_{k} = \sum_{k}^{} \qty(a_{k} \lambda_{k}) v_{k} \end{equation}

Insights to SVD: “ever matrix is a diagonal matrix, when viewed in the right space”

We can solve a linear system by moving it around:

\begin{align} Ac = b \\ \Rightarrow\ & U \Sigma V^{T} c = b \\ \Rightarrow\ & U \qty(\Sigma V^{T} c) = b \\ \Rightarrow\ & \Sigma V^{T} c = U^{T} b \end{align}

(since \(U\) is orthonormal, we can just flip it to invert it)

Call \(U^{T} b = \hat{b}\), call \(V^{T} c = \hat{c}\). We now have: