Modality-Agnostic Representation Learning via Hierarchical Variational Auto-Encoders

Learning pixel-level modality-agnostic representation of multi-modal imaging data is a challenging and open problem. In this work, Dr. Reuben Dorent will introduce MHVAE, a deep hierarchical variational auto-encoder (VAE) that generates modality-agnostic representations at the pixel level and allows for missing imaging modalities at training and testing time. Extending multi-modal VAEs with a hierarchical latent structure, a parametrization of the approximate posterior is introduced with a factorization similar to the true posterior, which can be expressed as a combination of unimodal variational posteriors. A simple optimization strategy is proposed to encourage learned representations to be modality-agnostic. Experiments on a database of intra-operative ultrasound (iUS) and Magnetic Resonance (MR) images demonstrate the effectiveness of the proposed approach at generating pixel-level representations that retain the content information while being similar for different sets of input modalities.