\begin{align} p_{\text{fail}} &= \mathbb{E}_{\tau \sim p\qty(\cdot)} \qty [1 \qty{\tau \not \in \psi}] \\ &= \int 1 \qty {\tau \not\in \psi} p\qty(\tau) \dd{\tau } \end{align}
that is, the Probability of Failure is just the normalizing constant of the Failure Distribution. Like with Failure Distribution itself, computing this is quite intractable. We have a few methods to solve this, namely:
- direct estimation: directly approximate your failure probability from nominal distribution \(p\) — \(\tau_{j} \sim p\qty(\cdot)\), \(\hat{p}_{\text{fail}} = \frac{1}{m} \sum_{i=1}^{m} 1\qty {\tau_{i} \not \in \psi}\)
- Importance Sampling: design a distribution to probe failure, namely proposal distribution \(q\), and then reweight by how different it is from \(p\) — \(\tau_{j} \sim q\qty(\cdot)\), \(\hat{p}_{\text{fail}} = \frac{1}{m}\sum_{i=1}^{m} w_{i} \mathbb{1} \qty {\tau_{i}\not \in \psi}\), call \(w_{i} = \frac{p\qty(\tau_{i})}{q\qty(\tau_{i})}\) (the “importance weight”)
- adaptive importance sampling
- multiple importance sampling
- sequential monte-carlo
How do you pick a proposal distribution? See proposal distribution.
evaluating estimators
- bias: an estimator is unbiased if it predicts the true value in expectation; namely \(\mathbb{E}\qty [\hat{p}_{\text{fail}}] = p_{\text{fail}}\)
- consistency: an estimator is consistent if it converges to infinity at infinite samples, namely that \(\text{lim}_{m \to \infty} \hat{p}_{\text{fail}} = p_{\text{fail}}\)
- variance: we want the variance of the estimates around the true value to be low
evaluation of direct estimation
\begin{equation} \text{Var}\qty [\hat{p}] = \frac{p_{\text{fail}} \qty(1-p_{\text{fail}})}{m} \end{equation}
- as the number of samples increases, the variance in the estimate decreases (yay!)
- as the probability of failure deceases, the variance in the failure actually icreases!!! baaaad vibes!