effability

see also language

Transitions

1. Transition first from rule based learning to statistical learning
2. Rise of semantic parsing: statistical models of parsing
3. Then, moving from semantic parsing to large models—putting decision making and language modeling into the same bubble

Importance of LLMs

• They are simply better at understanding language inputs
• They can generate structured information (i.e. not just human language, JSONs, etc.)
• They can perform natural language “reasoning”—not just generate

(and natural language generation, abv)

What makes language modeling hard: resolving ambiguity is hard.

“the chef made her duck”

Contents

Basic Text Processing

• regex
• ELIZA
• tokenization and corpus
  • Herdan’s Law
• text normalization
  • tokenization + Subword Tokenization
    • BPE
  • Word Normalization
    • lemmatization through morphological parsing
    • only take stems from morphemes: porter stemmer
  • sentence segmentation
• N-Grams

Edit Distance

DP costs \(O(nm)\), backtrace costs \(O(n+m)\).

• minimum edit distance
  • weighted edit distance
• backtracing

Ngrams

• N-Grams
• Markov Assumption
  • Unigrams
  • Backoff and Stupid Backoff
  • Interpolation
  • OOV Words
• Model Evaluation
  • perplexity
  • open vocabulary

Text Classification

• Text Classification
  • Bag of Words
  • Naive Bayes
  • Naive Bayes for Text Classification
    • Binary Naive Bayes
  • Lexicon
• Naive Bays Language Modeling
• Harmonic Mean
  • Macroaverage and Microaverage

Logistic Regression

• Generative Classifier vs Discriminate Classifier
• Logistic Regression Text Classification
  • decision boundary
• cross entropy loss
• stochastic gradient descent

Information Retrial

• Information Retrival
  • Term-Document Matrix
  • Inverted Index + postings list
    • Boolean Retrieval
    • positional index

Ranked Information Retrial

• Ranked Information Retrieval
  • feast or famine problem
  • free text query
• score
  • Jaccard Coefficient
  • log-frequency weighting
  • document frequency ("idf weight")
  • TF-IDF
    • SMART notation
• vector-space model

Vector Semantics

• sense
  • principle of contrast
  • word relatedness
    • semantic field
  • synonymy and antonyms
  • affective meaning
• vector semantics
  • transposing a Term-Document Matrix
  • term-term matrix
  • word2vec
    • skip-gram with negative sampling

POS and NER

• POS Tagging
• NER Tagging

Dialogue Systems

• Dialogue
• Chatbot
  • PARRY

Recommender Systems

• Recommender System

Dora

• Dora

Neural Nets

• Neural Networks

The Web

• Web Graph
• Social Network

A machine learning model: input — last n words, output — probabilist distribution over the next word. An LM predicts this distribution (“what’s the distribution of next word given the previous words):

\begin{equation} W_{n} \sim P(\cdot | w^{(t-1)}, w^{(t-2)}, \dots, w^{(1)}) \end{equation}

By applying the chain rule, we can also think of the language model as assigning a probability to a sequence of words:

\begin{align} P(S) &= P(w^{(t)} | w^{(t-1)}, w^{(t-2)}, \dots, w^{(1)}) \cdot P(w^{(t-1)} | w^{(t-2)}, \dots, w^{(1)}) \dots \\ &= P(w^{(t)}, w^{(t-1)}, w^{(t-2)}, \dots, w^{(1)}) \end{align}