A permutation \(\pi\) of some \(\{1,2,…, n\}\) is a rearrangement of this list. There are \(n!\) different permutations of this set.

A permutation is an ORDERED arrangement of objects.

permutation with indistinct objects

What if you want to order a set with sub-set of indistinct objects? Like, for instance, how many ways are there to order:

\begin{equation} 10100 \end{equation}

For every permutation of \(1\) in this set, there are two copies being overcounted.

Suppose you have a series of column pivots:

\begin{equation} \qty(P_{r_2} M_{11} P_{r_1} \dots) \end{equation}

you can combine all row permutations and column permutations down

perplexity is a measure of a language model’s ability to predict words.

Intuition

A good language model should prefer “real” or otherwise “frequently observed” sentences. That is, it should assign lower probability to word salad.

So a good language model should assign a higher probability to the next word that actually occurs given a sequence of words.

Generally, we want the LM to assign high probability to the entire test set. However, big issue is that probability gets smaller by length of the text.

PET is a type of plastic.

Consider a family of bacterial:

\begin{equation} P’ = 2P \end{equation}

this is a normal exponential growth situation. However, we know this isn’t true. Because the nutrients in the petri dish has a finite amount of nutrients. Hopefully this rule succeeds when the population is small, and should stop when the growth is bounded.

For instance, say you can never have more than 100 bacteria:

\begin{equation} P’ = 2P(100-P) \end{equation}

See logistic equation for solution