sigmoid function is used to squash your data between \(0\) and \(1\). Sigmoid is symmetric. It could take any number and squash it to look like a probability between 0 and 1.

\begin{equation} \sigma(z) = \frac{1}{1+ e^{-z}} \end{equation}

Say you have one discrete variable \(X\), and one continuous variable \(Y\), and you desire to express \(p(x|y)\).

The simplest way to do this, of course, is to say something like:

\begin{equation} P(x^{j} \mid y) = \begin{cases} P(x^{j} \mid y) = 0, y < \theta \\ P(x^{j} \mid y) = 1, y > \theta \end{cases} \end{equation}

Some Ideas

• Error Correction Codes
• Sampling + Quantization
• Compression Algorithms
• Frequency Domain Technologies

Two Main Goals

1. Unit 1: Efficient Representation of Signal (i.e. compression)—we ideally want the smallest sequence of bits to encode the raw signal
2. Unit 2: Preserving Information of Signal (i.e. communication)—we ideally want to communicate our bits while not sacrificing information despite all communication channels being noisy

Unit 1 outline

• compress the same exactly information into less space (lossless compression)
  • what is information (probability and entropy)
  • compression and limits of compression (Huffman Coding)
• removing irrelevant/uninteresting information (lossy compression)
  • key idea: “frequency domain can be aggressively compressed”
  • signals, frequency representation, bandwidth (discrete cosine transform)
  • quantization, sampling, reconstruction (encoding analog signal into digital signal)

Unit 2 outline

• communication basics (channels and noise)
• representing bits for physical/analogue communication (modulation—encoding digital signal into analog signal)
• bandwidth, spectrum shaping/sharing (frequency-domain filtering)
• fundamental limits (channel capacity)
• separation of compression and communication (separation principle)
• adding redundancy to communication schemes (error-correcting codes)

Lectures

Unit 1

SU-ENGR76 Unit 1 Index

Here is the most simple Differential Equation one could imagine:

\begin{equation} \dv{x}{t} = f(t,x) \end{equation}

Or, perhaps, we have a second order differential equation which is the same thing but in the second degree:

\begin{equation} \dv[2]{x}{t} = f\qty(t,x,\dv{x}{t}) \end{equation}

Then in which case, we have that the first most simple type of differential equation to be as follows:

\begin{equation} \dv{x}{t} = x(t) \end{equation}

If we can solve this, we can generalize this to most of other First-Order Differential Equations.

Uses a randomness temperature, which starts high for exploration and slowly decreases by one of the Simulated Annealing Schedules. Whenever you are about to take a step, each step must be “accepted” with probability given by the Metropolis Criteria.

Intuition: “sometimes, you try something that’s bad, to jump out of local optima.”

Metropolis Criteria

\begin{align} P(\text{accept}) = \begin{cases} 1, \text{if}\ \Delta y \leq 0\\ \min \qty(e^{-\frac{\Delta y}{t}}, 1), \text{otherwise} \end{cases} \end{align}

where \(t\) is the temperature; meaning—-even if the next point \(\Delta y > 0\) (meaning its bad, because it increases loss), we sometimes take that ponit still.