Take X-Rays and generate clinical reports.

Method

encoder decoder architectures

Encoder

ConViT: convolutional vision transformer. Special thing: we swap out the attention

Double Weighted Multi-Head Attention

We want to force the model to focus on one thing, so we modulate the model based on the weights of other: if one head is big, we make the other head small.

where \(w_{\cos i} = \frac{\sum_{i}^{} \cos \qty (att_{a}, att_{base})}{N}\)

\begin{equation} w = w_{a} \cdot (1- w_{\cos i}) \end{equation}

simple games

constituents

• agent \(i \in X\) the set of agents.
• joint action space: \(A = A’ \times A^{2} \times … \times A^{k}\)
• joint action would be one per agent \(\vec{a} = (a_{1}, …, a_{k})\)
• joint reward function \(R(a) = R’(\vec{a}), …, R(\vec{a})\)

additional information

prisoner’s dilemma

traveler’s dilemma

• two people write down the price of their luggage, between 2-100
• the lower amount gets that value plus 2
• the higher amount gets the lower amount minus 2

joint policy agent utility

for agent number \(i\)

Key idea: multi-modality, when leveraged well, leads to faster convergence.

Data Availability

Health and health sensing requires labels, but health signals require specialist knowledge + broader context to label.

• typical image labeling: 0.05/label
• medical imaging: 4.00/label

Even if want to automate the study, we need to Kyntic style strap a thing to a person and have soft labels that we align with raw sensor data..

Instead, Do Time-series

Instead: run proxy self-supervised studies into the future—pretraining on a shit tone of sensor data just as timeseries regressing into the future without any labels.

Its a general form of the combinations formula:

\begin{equation} {n \choose k_1, k_2, \dots, k_{n}} = \frac{n!}{k_{1}! k_2! \dots k_{n}!} \end{equation}

what if we did Importance Sampling, but…. had multiple proposals?!

notation: \(w_{i}, \tau_{i}\), etc. all correspond to stuff that came from proposal \(q_{i}\).

standard multiple importance sampling (s-MIS)

1. draw samples from current proposals \(\tau_{i} \sim q_{i}\qty(\tau)\)
2. use all of the samples to estimate \(p_{\text{fail}}\)

\begin{equation} \hat{p}_{\text{fail}} = \frac{1}{m} \sum_{i=1}^{m} w_{i} 1\qty {\tau_{i}\not \in \psi} \end{equation}

where

\begin{equation} w_{i} = \qty(\frac{p\qty(\tau_{i})}{q_{i}\qty(\tau_{i})}) \end{equation}

deterministic mixture multiple importance sampling (DM-MIS)

1. draw samples alternating each of the proposals
2. use them to estimate \(p_{\text{fail}}\)

\begin{equation} w_{i} = \frac{p\qty(\tau_{i})}{\frac{1}{m}\sum_{j=1}^{m}q_{j}\qty(\tau_{i})} \end{equation}