InVA: Integrative Variational Autoencoder for Harmonization of Multi-modal Neuroimaging Data

2024-02-05

arXiv.org

AI Summary

Positron emission tomography (PET) scans—critical for neuroimaging—are costly and involve ionizing radiation, limiting their clinical and research accessibility. Structural magnetic resonance imaging (sMRI), while widely available and safe, lacks the molecular specificity of PET. Method: To address this, we propose an integrative variational autoencoder (iVAE), a novel deep generative model that explicitly models the strong nonlinear mapping between sMRI and PET. iVAE jointly embeds multimodal features, constructs a shared latent space, and performs end-to-end optimization—enabling explicit modeling of inter-modal dependencies, unlike conventional VAEs that treat inputs independently. Contribution/Results: Evaluated on real brain imaging data, iVAE significantly outperforms standard VAEs in PET synthesis, yielding higher-fidelity, clinically meaningful reconstructions. Its interpretable architecture and scalability establish a new paradigm for low-cost, radiation-free neuroimaging alternatives—bridging the gap between anatomical and functional imaging without requiring PET acquisition.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

There is a significant interest in exploring non-linear associations among multiple images derived from diverse imaging modalities. While there is a growing literature on image-on-image regression to delineate predictive inference of an image based on multiple images, existing approaches have limitations in efficiently borrowing information between multiple imaging modalities in the prediction of an image. Building on the literature of Variational Auto Encoders (VAEs), this article proposes a novel approach, referred to as Integrative Variational Autoencoder ( exttt{InVA}) method, which borrows information from multiple images obtained from different sources to draw predictive inference of an image. The proposed approach captures complex non-linear association between the outcome image and input images, while allowing rapid computation. Numerical results demonstrate substantial advantages of exttt{InVA} over VAEs, which typically do not allow borrowing information between input images. The proposed framework offers highly accurate predictive inferences for costly positron emission topography (PET) from multiple measures of cortical structure in human brain scans readily available from magnetic resonance imaging (MRI).

Problem

• Predicting costly PET scans from structural MRI
• Modeling outcome images using shared and modality-specific features
• Integrating multiple imaging modalities for neuroimaging research

Innovation

• Hierarchical VAE framework for multimodal neuroimaging
• Models outcome images with shared and specific features
• Predicts PET scans from structural MRI efficiently