Unified Cross-Modal Image Synthesis with Hierarchical Mixture of Product-of-Experts

2024-10-25

arXiv.org

AI Summary

To address the cross-modal high-resolution synthesis of preoperative MRI and intraoperative ultrasound (US) images, this paper proposes a Hierarchical Mixture-of-Experts Variational Autoencoder (MMHVAE). The method tackles key challenges including multimodal latent distribution modeling, missing modality estimation, training with incomplete data, and effective information fusion. Technically, it introduces— for the first time—a hierarchical mixture-of-experts architecture to jointly model multimodal latent spaces; incorporates explicit variational inference to impute missing modalities and integrates dataset-level priors to enhance robustness; and synergistically combines Product-of-Experts (PoE) fusion with cross-modal latent alignment to achieve unified representation of multiparametric MRI and US. Evaluated on brain imaging data, MMHVAE achieves significant improvements in PSNR and SSIM. Synthesized images exhibit accurate anatomical structures and sharp textural details, enabling real-time intraoperative navigation.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Mixture of Experts (MoE)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We propose a deep mixture of multimodal hierarchical variational auto-encoders called MMHVAE that synthesizes missing images from observed images in different modalities. MMHVAE's design focuses on tackling four challenges: (i) creating a complex latent representation of multimodal data to generate high-resolution images; (ii) encouraging the variational distributions to estimate the missing information needed for cross-modal image synthesis; (iii) learning to fuse multimodal information in the context of missing data; (iv) leveraging dataset-level information to handle incomplete data sets at training time. Extensive experiments are performed on the challenging problem of pre-operative brain multi-parametric magnetic resonance and intra-operative ultrasound imaging.

Problem

• Synthesizing missing images across different modalities
• Creating complex latent representations for high-resolution generation
• Learning multimodal fusion with incomplete training datasets

Innovation

• Hierarchical mixture of multimodal variational autoencoders
• Generates missing images from observed multimodal data
• Learns latent representations for cross-modal image synthesis