New Concepts

• structure learning
  • K2 Algorithm
  • Local Search
• Markov Equivalence Classes
• Partially Directed Graph Search
• utility and Rational Preferences
  • lottery
  • utility elicitation
  • expected utility
  • risk aversion
• decision network
• utility functions
  • quadratic utility
  • exponential utility
  • power utility and its special case log utility
• maximum expected utility principle
• value of information

Important Results / Claims

• checking Markov Equivalence
• utility of Rational Preference
• von Neumann and Morgenstern Axioms: Axioms for checking rationality
• never have a utility function that’s infinite
• utility of a lottery
• process of observation selection

Questions

Notation

“state variables” represent the contents of the state; “state” is a complete assignment of state variables.

New Concepts

• Markov Decision Process
  • stationary Markov Decision Process
  • finite-horizon models + infinite-horizon models
• policy
  • stationary policy
  • optimal policy and optimal value function
  • policy evaluation and policy iteration
    • lookahead equation
    • Bellman Expectation Equation
    • action-value function and value-function policy
    • advantage function

Important Results / Claims

• policy evaluation methods
  • solving for the utility of a policy
  • finding the best policy
• policy iteration

Questions

• why is it d seperated

Interesting Factoids

Key Sequence

Notation

New Concepts

• Markov Decision Process
  • value iteration
• Bellman Residual
• for continuous state spaces: Approximate Value Function
  • use global approximation or local approximation methods

Important Results / Claims

• policy and utility
  • creating a good utility function / policy from instantaneous rewards: either policy evaluation or value iteration
  • creating a policy from a utility function: value-function policy (“choose the policy that takes the best valued action”)
  • calculating the utility function a policy currently uses: use policy evaluation
• kernel smoothing
• value iteration, in practice

Questions

Interesting Factoids

Key Sequence

Notation

New Concepts

• global approximation
• online planning
  • Rollout with Lookahead
  • Forward Search
  • Branch and Bound
  • monte-carlo tree search
• open loop planning

Important Results / Claims

• Rollout Policy
• monte-carlo exploration
• open-loop planning vs close-loop planning

Questions

Interesting Factoids

Big day. Policy Gradient.

New Concepts

• Approximate Policy Evaluation and Roll-out utility
• Policy Optimization methods:
  • Local Policy Search (aka Hooke-Jeeves Policy Search)
  • Genetic Policy Search
  • Cross Entropy Method
  • Policy Gradient, Regression Gradient and Likelyhood Ratio Gradient
  • Reward-to-Go

Important Results / Claims

• monte-carlo policy evaluation
• Finite-Difference Gradient Estimation
• Linear Regression Gradient Estimate

Questions for next office hour