Take X-Rays and generate clinical reports.
Method
encoder decoder architectures
Encoder
ConViT: convolutional vision transformer. Special thing: we swap out the attention
Double Weighted Multi-Head Attention
We want to force the model to focus on one thing, so we modulate the model based on the weights of other: if one head is big, we make the other head small.
where \(w_{\cos i} = \frac{\sum_{i}^{} \cos \qty (att_{a}, att_{base})}{N}\)
\begin{equation} w = w_{a} \cdot (1- w_{\cos i}) \end{equation}
meaning:
\begin{equation} att_{dwma} = w \cdot att \end{equation}
Decoding
Goood ol’ Hierarchical-Decoder