Key Sequence

Notation

New Concepts

• Restricted Gradient and Actor-Critic
• Exploration and Exploitation with Binary Bandit
  • Undirected Exploration (Explore-then-commit)
  • Directed Exploration (Softmax Method, Quantile Exploration, UCB 1, Posterior Sampling)

Important Results / Claims

• Bayesian Model Estimation and greedy action
• epsilon-greedy exploration with decay

Questions

Interesting Factoids

See Double Envelope Problem

Key Sequence

• we introduced Decision Making, models of decision making and gave some examples
• we introduced different types of uncertainty
• in the reading, we introduced the Course Outline

New Definitions

• Decision Making
• uncertainty

Questions for Jana

• how are planning methods different from explicit programming methods?
• “Sequential Decision Process”
• Partially observable Markov decision process (POMDP)

definitions

• probability and random variable
  • uniform distribution
  • Gaussian distribution
• probability distributions
• conditional probability
• Bayes Theorem

results

• axiom of probability

support = range

Notation

shorthand for probability

Take

\begin{equation} P(X = 1) = \frac{1}{6} \end{equation}

We can write this in short hand like:

\begin{equation} P(X^{1}) = P(X=1) = \frac{1}{6} \end{equation}

\(P\) vs \(p\)

Upper case \(P\) for probability mass function (one shot chance), lower case \(p\) for probability density functions (integral)

New Concepts

• degrees of belief and describing them using the language of probability
• discrete distribution and continuous distribution and joint probability distribution
  • important tools:
    • parameters of a distribution
    • probability density functions
    • cumulative distribution function
    • quantile function
• fun probability distributions
  • Gaussian distribution + Truncated Gaussian distribution
  • uniform distribution
• conditional probability and Bayes Theorem
  • unique models that leverage conditional probability
    1. conditional Gaussian models
    2. linear gaussian model
    3. conditional linear Gaussian models: use your big brain to add up 1) and 2), with continuous random variables \(X, Y\), and a discrete \(Z\), where \(p(x \mid y, z)\).
    4. sigmoid model
• Baysian Network and conditional independence
  • d seperation

Important Results / Claims

• history and impact of decision making
• law of total probability
• fun axioms
  • belief axioms:
    • universal comparability
    • transitivity
  • probability axioms:
    • axiom of probability
• Methods of Compressing the Parameters of a Distribution
  • assuming independence
  • using a decision tree
• checking for conditional independence

Questions