Maximum Likelihood Parameter Learning
Last edited: August 8, 2025“We find the parameter that maximizes the likelihood.”
- for each \(X_{j}\), sum
- what’s the log-likelihood of one \(X_{i}\)
- take derivative w.r.t. \(\theta\) and set to \(0\)
- solve for \(\theta\)
(this maximizes the log-likelihood of the data!)
that is:
\begin{equation} \theta_{MLE} = \arg\max_{\theta} P(x_1, \dots, x_{n}|\theta) = \arg\max_{\theta} \qty(\sum_{i=1}^{n} \log(f(x_{i}|\theta)) ) \end{equation}
If your \(\theta\) is a vector of more than \(1\) thing, take the gradient (i.e. partial derivative against each of your variables) of the thing and solve the place where the gradient is identically \(0\) (each slot is \(0\)). That is, we want:
MaxQ
Last edited: August 8, 2025Two Abstractions
- “temporal abstractions”: making decisions without consideration / abstracting away time (MDP)
- “state abstractions”: making decisions about groups of states at once
Graph
MaxQ formulates a policy as a graph, which formulates a set of \(n\) policies

Max Node
This is a “policy node”, connected to a series of \(Q\) nodes from which it takes the max and propegate down. If we are at a leaf max-node, the actual action is taken and control is passed back t to the top of the graph
MBP
Last edited: August 8, 2025MCVI
Last edited: August 8, 2025MCVI solves POMDPs with continuous state space, but with discrete observation and action spaces. It does this by formulating a POMDP as a graph.
Fast algorithms require discretized state spaces, which makes the problem much more difficult to model. MCVI makes continuous representations possible for complex domains.
MC Backup
Normal POMDP Bellman Backup isn’t going to work well with continuous state spaces.
Therefore, we reformulate our value backup as:
\begin{equation} V_{t+1}(b) = \max_{a \in A} \qty(\int_{s} R(s,a)b(s) \dd{s}) + \gamma \sum_{o \in O}^{} p(o|b,a) V_{t}(update(b,a,o)) \end{equation}
