see also skip-gram with negative sampling

sample some negative samples and negate them

Neoclassical Economics is a view of economics that disregards the Keynsian Politics theory of the economy needs a minder started by Milton Freedman. It believes that free market economy will prevail.

while POS Tagging assigns tags to each word, NER Tagging tags the category of usage of multi-word spans.

NER Tagging needs to label spans of text, which means that there is ambiguity in type.

BIO Tagging

BIO Tagging will tag each word: where \(B\) begins a span, \(I\), is inside a span, and \(O\) outside a span. So tags per word still apply, but we can extract span information as well.

(job - gender + gender ) = job (captial - country + country) = captial

We can think of a neural network as a roll-out of a system. For ReLU networks in particular, we can compute the exact reachable set!

Suppose we have the input set \(s_1\); let’s consider:

\begin{equation} z_1 = W_1 s_1 + b_1 \end{equation}

after one linear layer. We can then apply a nonlinear function to it. The beauty with ReLU nonlinearities is that we can split our network into one set per quadrant, and consider what ReLU will do to it.

Neural Network Unit

A real-valued vector as input, each multiplied by some weights, summed, and squashed by some non-linear transform.

\begin{equation} z = w\cdot x + b \end{equation}

and then, we will squash this using it as an “activation”

\begin{equation} y = \sigmoid(z) \end{equation}

One common activation is sigmoid. So, one common formulation would be:

\begin{equation} y = \frac{1}{1+\exp (- (w \cdot x + b))} \end{equation}

Tanh

\begin{equation} y(z) = \frac{e^{z} - e^{-z}}{e^{z}+e^{-z}} \end{equation}