Naive Bayes is a special class of Baysian Network inference problem which follows a specific structure used to solve classification problems.

The Naive Bayes classifier is a Baysian Network of the shape:

(Why is this backwards(ish)? Though we typically think about models as a function M(obs) = cls, the real world is almost kind of opposite; it kinda works like World(thing happening) = things we observe. Therefore, the observations are a RESULT of the class happening.)

constituents

• \(\mu\) the mean
• \(\sigma\) the variance

requirements

\begin{equation} X \sim N(\mu, \sigma^{2}) \end{equation}

Its PDF is:

\begin{equation} \mathcal{N}(x \mid \mu, \sigma^{2}) = \frac{1}{\sigma\sqrt{2\pi}} e^{ \frac{-(x-u)^{2}}{2 \sigma^{2}}} \end{equation}

where, \(\phi\) is the standard normal density function

Its CDF:

\begin{equation} F(x) = \Phi \qty( \frac{x-\mu}{\sigma}) \end{equation}

We can’t integrate \(\Phi\) further. So we leave it as a special function.

And its expectations:

\(E(X) = \mu\)

\(Var(X) = \sigma^{2}\)

additional information

multi-variant Gaussian density

\begin{equation} z \sim \mathcal{N} \qty(\mu, \Sigma) \end{equation}

Gaussian Discriminant Analysis

High level idea: 1) fit parameters to the positive and negative examples separately as a multi-variant Gaussian density 2) try to see if a new samples’ probablitity to 1) is greater or 2) is greater.

requirements

• fit a parameter to

additional information

multivariant gaussian

See multi-variant Gaussian density. If it helps, here you go:

\begin{equation} p\qty(z) = \frac{1}{\qty(2\pi)^{\frac{|z|}{2}}|\Sigma|^{\frac{1}{2}}} \exp \qty(-\frac{1}{2} \qty(x-\mu)^{T} \Sigma^{-1} \qty(x - \mu)) \end{equation}

making predictions

Suppose \(p\qty(y=1) = \phi\), \(p\qty(y=0) = 1-\phi\).

constituents

• \(x \in \qty {0,1}^{m}\) for \(m\) features
• \(y \in \qty {0,1}\) for labels
• \(\phi_{j|y=1}\), for \(p\qty(x_{j} = 1 | y=1)\)
• \(\phi_{j|y=0}\), for \(p\qty(x_{j} = 1 | y=0)\)
• \(\phi_{y}\), for \(p\qty(y=1)\), the “cost prior”

assumption

ASSUME: features in \(x\) are conditionally independent given \(y\)

That is, we assume that:

\begin{align} p\qty(x|y) &= p\qty(x_1, x_2, \dots, x_{1000} | y) \\ &= p\qty(x_1|y) p\qty(x_2|y, x_1) p\qty(x_3|y, x_1, x_2) \dots p\qty(x_{1000}|y, x_1, \dots, x_{999}) \end{align}

Key Sequence

Notation

New Concepts

• multi-variant Gaussian density
• Generative Learning Algorithm

Important Results / Claims

• applying Bayes rule to make prediction makes the sigmoid function

Generative and Discriminate Algorithm

Discriminative Algorithm

Learns \(p\qty(y|x)\), or better yet just \(h\qty(x) \in \qty {0,1}\).

“what’s the distribution over labels”

Generative Algorithm

Learns \(p\qty(x|y)\) as well as a “class prior” \(p\qty(y)\). To make classification decisions, you apply Bayes rule. THINK: Naive Bayes.