Big Data is a term for datasets large enough that traditional data processing applications are inadequate. i.e. when non-parallel processing is inadequate.

That is: “Big Data” is when Pandas and SQL is inadequate. To handle big data, its very difficult to sequentially go through and process stuff. To make it work, you usually have to perform parallel processing under the hood.

Rules of Thumb of Datasets

• 1000 Genomes (AWS, 260TB)
• CommonCraw - the entire web (On PSC! 300-800 TB)
• GDELT - https://www.gdeltproject.org/ a dataset that contains everything that’s happening in the world right now in terms of news (small!! 2.5 TB per year; however, there is a LOT of fields: 250 Million fields)

Evolution of Big Data

Good Ol’ SQL

1. schemas are too set in stone (“not a fit for Agile development” — a research scientist)
2. SQL sharding, when working correctly, is

KV Stores

And this is why we gave up and made Redis (or Amazon DynamoDB, Riak, Memcached) which keeps only Key/Value information. We just make the key really really complicated to support structures: GET cart:joe:15~4...

Intuition:

• \(O\): \(\leq\)
• \(\theta\): \(=\)
• \(\Omega\): \(\geq\)

Definitions:

• \(f(n) = O(g(n)) \implies \exists n_{0}: \forall n > n_0, f(n) \leq c (g(n))\)
• \(f(n) = \Omega(g(n)) \implies \exists n_{0}: \forall n > n_0, f(n) \geq c (g(n))\)
• \(f(n) = \theta(g(n)) \implies \exists n_{0}: \forall n > n_0, f(n) \geq 1 (g(n)), f(n) \leq c (g(n))\)

Little ones:

• \(o\): \(<\)
• \(\omega\): \(>\)

A binary operation means that you are taking two things in and you are getting one thing out; for instance:

\begin{equation} f: (\mathbb{F},\mathbb{F}) \to \mathbb{F} \end{equation}

This is also closed, but binary operations doesn’t have to be.

A binomial distribution is a typo of distribution whose contents are:

• Binary
• Independent
• Fixed number
• Same probability: “That means: WITH REPLACEMENT”

Think: “what’s the probability of \(n\) coin flips getting \(k\) heads given the head’s probability is \(p\)”.

constituents

We write:

\begin{equation} X \sim Bin(n,p) \end{equation}

where, \(n\) is the number of trials, \(p\) is the probability of success on each trial.

requirements

Here is the probability mass function:

\begin{equation} P(X=k) = {n \choose k} p^{k}(1-p)^{n-k} \end{equation}