_index.org

Maximum Likelihood Parameter Learning

Last edited: August 8, 2025

“We find the parameter that maximizes the likelihood.”

  1. for each \(X_{j}\), sum
    1. what’s the log-likelihood of one \(X_{i}\)
  2. take derivative w.r.t. \(\theta\) and set to \(0\)
  3. solve for \(\theta\)

(this maximizes the log-likelihood of the data!)

that is:

\begin{equation} \theta_{MLE} = \arg\max_{\theta} P(x_1, \dots, x_{n}|\theta) = \arg\max_{\theta} \qty(\sum_{i=1}^{n} \log(f(x_{i}|\theta)) ) \end{equation}


If your \(\theta\) is a vector of more than \(1\) thing, take the gradient (i.e. partial derivative against each of your variables) of the thing and solve the place where the gradient is identically \(0\) (each slot is \(0\)). That is, we want:

MaxQ

Last edited: August 8, 2025

Two Abstractions

  • “temporal abstractions”: making decisions without consideration / abstracting away time (MDP)
  • “state abstractions”: making decisions about groups of states at once

Graph

MaxQ formulates a policy as a graph, which formulates a set of \(n\) policies

Max Node

This is a “policy node”, connected to a series of \(Q\) nodes from which it takes the max and propegate down. If we are at a leaf max-node, the actual action is taken and control is passed back t to the top of the graph

MBP

Last edited: August 8, 2025

MCVI

Last edited: August 8, 2025

MCVI solves POMDPs with continuous state space, but with discrete observation and action spaces. It does this by formulating a POMDP as a graph.

Fast algorithms require discretized state spaces, which makes the problem much more difficult to model. MCVI makes continuous representations possible for complex domains.

MC Backup

Normal POMDP Bellman Backup isn’t going to work well with continuous state spaces.

Therefore, we reformulate our value backup as:

\begin{equation} V_{t+1}(b) = \max_{a \in A} \qty(\int_{s} R(s,a)b(s) \dd{s}) + \gamma \sum_{o \in O}^{} p(o|b,a) V_{t}(update(b,a,o)) \end{equation}

meal replacement

Last edited: August 8, 2025