Key Sequence

Notation

New Concepts

• Markov Decision Process
  • value iteration
• Bellman Residual
• for continuous state spaces: Approximate Value Function
  • use global approximation or local approximation methods

Important Results / Claims

• policy and utility
  • creating a good utility function / policy from instantaneous rewards: either policy evaluation or value iteration
  • creating a policy from a utility function: value-function policy (“choose the policy that takes the best valued action”)
  • calculating the utility function a policy currently uses: use policy evaluation
• kernel smoothing
• value iteration, in practice

Questions

Interesting Factoids

Key Sequence

Notation

New Concepts

• global approximation
• online planning
  • Rollout with Lookahead
  • Forward Search
  • Branch and Bound
  • monte-carlo tree search
• open loop planning

Important Results / Claims

• Rollout Policy
• monte-carlo exploration
• open-loop planning vs close-loop planning

Questions

Interesting Factoids

Big day. Policy Gradient.

New Concepts

• Approximate Policy Evaluation and Roll-out utility
• Policy Optimization methods:
  • Local Policy Search (aka Hooke-Jeeves Policy Search)
  • Genetic Policy Search
  • Cross Entropy Method
  • Policy Gradient, Regression Gradient and Likelyhood Ratio Gradient
  • Reward-to-Go

Important Results / Claims

• monte-carlo policy evaluation
• Finite-Difference Gradient Estimation
• Linear Regression Gradient Estimate

Questions for next office hour

Key Sequence

Notation

New Concepts

• Restricted Gradient and Actor-Critic
• Exploration and Exploitation with Binary Bandit
  • Undirected Exploration (Explore-then-commit)
  • Directed Exploration (Softmax Method, Quantile Exploration, UCB 1, Posterior Sampling)

Important Results / Claims

• Bayesian Model Estimation and greedy action
• epsilon-greedy exploration with decay

Questions

Interesting Factoids