multiprocessing is the act of switching between multiple processes so fast that it appears multiple processes are running concurrently.

• OS schedules tasks
• each program gets a little time, then has to wait in a turn to continue executing

base level syscalls that requires waiting will be moved off before finishing, and in the meantime others can wait. like file read.

program

A program is a script to be ran.

process

a process is an instance of a program. Every process has a unique identifier, each process is uniquely identified by a PID.

• we can have concurrency within a single process—each running a single function

We will solve problems:

• never race condition
• never deadlock

thread

• you can spawn a thread using the thread() can even pass function parameters
• threads share all virtual address space: bugs can arise when multiple threads modify the same thing at the same time—each thread has access to a small chunk of the stack
• threads are actually the unit of concurrency: the OS actually chooses threads to run

// now the thread can execute at any time: once a thread is made, it will run in any order
thread myThread(function_to_run, arg1, arg2, ...);
// threads run AS SOON AS SPAWENED: so

We can wait for a thread:

mutual information a measure of the dependence of two random variables in information theory. Applications include collocation extraction, which would require finding how two words co-occur (which means one would contribute much less entropy than the other.)

constituents

• \(X, Y\) random variables
• \(D_{KL}\) KL Divergence function
• \(P_{(X,Y)}\) the joint distribution of \(X,Y\)
• \(P_{X}, P_{Y}\) the marginal distributions of \(X,Y\)

requirements

mutual information is defined as

\begin{equation} I(X ; Y) = D_{KL}(P_{ (X, Y) } | P_{X} \otimes P_{Y}) \end{equation}

probability of “or”

If its not possible for two events to happen at the same time, they are called mutually exclusive:

\begin{equation} P(E\ or\ F) = P(E)+P(F) - P(E \cap F) \end{equation}

This is called the inclusion exclusion principle. This is what motivates inclusion exclusion counting.

General inclusion exclusion principle

Its scary. Think about this:

We basically need to alternate between adding and subtracting. (i.e.: in our case here, we add all the odd-group pairs (for P(x) and P(xyz)), we subtract the even-number pairs (for p(xy))).