Houjun Liu



the history: last states and actions \(h_{t} = (s_{1:t}, a_{1:t-1})\)



\begin{equation} a_{t} = \pi_{t}(h_{t}) \end{equation}

for a Markov Decision Process, our past states are d-seperated from our current action given knowing the state, so really we have \(\pi_{t}(s_{t})\)

Some policies can be stochastic:

\begin{equation} P(a_{t}) = \pi_{t}(a_{t} | h_{t}) \end{equation}

instead of telling you something to do at a specific point, it tells you what the probability it chooses of doing \(a_{t}\) is given the history.

additional information

stationary policy

For infinite-horizon models, our policy can not care about how many time stamps are left (i.e. we are not optimizing within some box with constrained time) and therefore we don’t really care about historical actions. So we have:

\begin{equation} \pi(s) \end{equation}

this can be used in infinite-horizon models against stationary Markov Decision Process.

optimal policy

\begin{equation} \pi^{*}(s) = \arg\max_{\pi} U^{\pi}(s) \end{equation}

“the most optimal policy is the policy that maximizes the expected utility of following \(\pi\) when starting from \(s\)”

We call the utility from the best policy the “optimal value function

\begin{equation} U^{*} = U^{\pi^{*}} \end{equation}

policy utility, and value

See policy evaluation