sample space \(S\) is the set of all possible outcomes of an experiment. It could be continuous or distinct.

equally likely outcomes

Some sample spaces have equally likely outcomes:

• coin flip
• flipping two coins
• rolling a fair die

If we have equally likely outcomes, \(P(outcome)\) = \(\frac{1}{S}\).

If your sample space has equally likely outcomes, the probability is juts counting:

\begin{equation} P(E) = \frac{count(E)}{count(S)} \end{equation}

Whenever you use this tool, you have to think about whether or not your outcomes are equally likely. For instance, the “sum of two dice rolling” is NOT equally likely.

SARS-COV2

1. traditional stain techniques to analyze the epitopes being targeted
2. uses cyro-EM structural analysis to figure structural points of neutralization
3. predict correct antibodies binding to force certain structures to neutralize covid-19
4. analyze mRNA-vax elicited antibodies to see similarity between those that are useful predicted in 3)

Study identified three epitopes: C1520, C1791, C1717, which changes the structure/activity of all three variants of concern as identified using methods above, and are inpervious to the mutation to the main supersite.

Sarsa (Lambda) is SARSA with Eligibility Traces (\(\lambda\)).

Previous approaches to deal with Partially Observable Markov Decision Process:

• memory-based state estimation (beliefs)
• special planning methods

Key question: Can we use MDP reinforcement learning to deal with POMDPs?

Background

Recall MDP SARSA:

\begin{equation} Q(s,a) \leftarrow Q(s,a) + \alpha \qty [(r + \gamma Q(s’, a’)) - Q(s,a)] \end{equation}

Recall that, sparse rewards with SARSA can take a long time to learn because it takes time to backpropgate.

Hence, we use Eligibility Traces, which keeps track of what’s “eligible” for updates:

Big problem: curse of dimensionality and the curse of history.

PBVI and HSVI tries to sample the belief simplex generally. But instead we should try to sample OPTIMAL REACHABLE SET.

Background

Recall one-step lookahead in POMDP. The difficulty here is that the sum over all of the alpha-vectors is still very hard. So, in PBVI, we only do this to a small set of beliefs

SARSOP

• sample \(R^{*}\)
• backup
• prune

Initialization

choose an initial belief, action, and observation using “suitable heuristics”. Initialize a set of alpha vectors corresponding to this belief.