Gaussian mixture model is a density estimation technique, which is useful for detecting out of distribution samples, etc.

We will use the superposition for a group of Gaussian distributions that would explain the dataset.

Suppose the data was generated from a Mixture of Gaussian; then for every data point \(x^{(i)}\) there is a latent \(z^{(i)}\) which tells you what Gaussian your data point is generated from.

So, for \(k\) Gaussian in your mixture:

\(z^{(i)} \in \qty {1, \dots, k}\) such that \(z^{(i)} \sim \text{MultiNom}\qty(\phi)\) (such that \(\phi_{j} \geq 0\), \(\sum_{j}^{} \phi_{j} = 1\))

Transpose Rules

• \(\qty(AB)^{T} = B^{T}A^{T}\)
• \(\qty(a^{T}Bc)^{T} = c^{T} B^{T}a\)
• \(a^{T}b = b^{T}a\)
• \(\qty(A+B)C = AC + BC\)
• \(\qty(a+b)^{T}C = a^{T}C + b^{T}C\)
• \(AB \neq BA\)

Derivative

Products

\begin{equation} \pdv{AB}{A} = B^{T}, \pdv{AB}{B} = A^{T} \end{equation}

\begin{equation} \pdv{Ax}{A} = x^{T}, \pdv{Ax}{x}= A \end{equation}

constituents

Let’s also define our entire training examples and stack them in rows:

\begin{equation} X = \mqty( - x^{(1)}^{T} - \\ \dots \\ - x^{\qty(n)}^{T} - ) \end{equation}

\begin{equation} Y = \mqty(y^{(1)} \\ \dots \\ y^{(n)}) \end{equation}

requirements

least-squares error becomes:

\begin{equation} J\qty(\theta) = \frac{1}{2} \sum_{i=1}^{n} \qty(h\qty(x^{(i)}) - y^{(i)}) ^{2} = \qty(X \theta - y)^{T} \qty(X \theta - y) \end{equation}

Solving this exactly by taking the derivative of \(J\) and set it to \(0\) (i.e. for a minima, we obtain)

Key Sequence

Notation

New Concepts

Important Results / Claims

Questions

Interesting Factoids

• log laws, exponent laws and general non-discrete math stuff

• distributions and infinite series, math53 content

• combinations

  • \(\mqty(n \\ k) = \mqty(n-1 \\ k-1) + \mqty(n-1 \\ k)\)
  • \(\mqty(n \\k) = \mqty(n \\ n-k)\)

• binomial theorem: \(\qty(a+b)^{n} = \sum_{k=0}^{n} \mqty(n \\k)a^{k} b^{n-k}\)

• geometric sum