Putting something with a different temperature in a space with a constant temperature. The assumption underlying here is that the overall room temperature stays constant (i.e. the thing that’s cooling is so small that it doesn’t hurt room temperature).

\begin{equation} y’(t) = -k(y-T_0) \end{equation}

where, \(T_0\) is the initial temperature.

The intuition of this modeling is that there is some \(T_0\), which as the temperature \(y\) of your object gets closer to t. The result we obtain

\begin{equation} \text{NL} = \text{NSPACE} \qty( \log n) \end{equation}

See also Certificates-Based Intepretation of NL

problems in \(NL\)

We can see \(L \subseteq NL\), because a TM is a NTM.

STCONN is in NL

On input \(\qty(G, s,t)\), if \(s = t\), accept; otherwise,

• currNode = 5
• numSteps = 0
• while steps <= n
  • non-deterministically choose a next node
  • update currNode = w
  • if w = t, accept
  • set numSteps ++
• reject

so we just have to remember the current node. So this whole thing is \(O\qty(\log n)\).

Complex System

Coherence

Generative REVOLUTION

Why probability maximization sucks

Its expensive!

Beam Search

1. Take \(k\) candidates
2. Expand \(k\) expansions for each of the \(k\) candidates
3. Choose the highest probability \(k\) candidates

\(k\) should be small: trying to maximizing

Branch and Bound

See Branch and Bound

Challenges of Direct Sampling

Direct Sampling sucks. Its sucks. It sucks. Just sampling from the distribution sucks. This has to do with the fact that assigning slightly lower scores “being less confident” is exponentially worse.

Learning Goals

1. Effective modern methods for deep NLP
   • Word vectors, FFNN, recurrent networks, attention
   • Transformers, encoder/decoder, pre-training, post-training (RLHF, SFT), adaptation, interoperability, agents
2. Big picture in HUMAN LANGUAGES
   • why are they hard
   • why using computers to deal with them are doubly hard
3. Making stuff (in PyTorch)
   • word meaning
   • dependency parsing
   • machine translation
   • QA

Lectures

• SU-CS224N APR022024
• SU-CS224N APR092024
• SU-CS224N APR112024
• SU-CS224N APR162024
• SU-CS224N APR182024
• SU-CS224N APR232024
• SU-CS224N APR252024
• SU-CS224N APR302024
• SU-CS224N MAY022024
• SU-CS224N MAY072024

• 1990 static word embeddings
• 2003 neural language models
• 2008 multi-task learning
• 2015 attention
• 2017 transformer
• 2018 trainable contextual word embeddings + large scale pretraining
• 2019 prompt engineering

Motivating Attention

Given a sequence of embeddings: \(x_1, x_2, …, x_{n}\)

For each \(x_{i}\), the goal of attention is to produce a new embedding of each \(x_{i}\) named \(a_{i}\) based its dot product similarity with all other words that are before it.

Let’s define: