Deep histological synthesis from mass spectrometry imaging for multimodal registration

2025-06-05

AI Summary

To address the cross-modal registration challenge between histopathology images and mass spectrometry imaging (MSI) data—arising from modality heterogeneity and dimensional disparity—this paper proposes a novel single-modality registration paradigm based on generative synthesis: the first end-to-end MSI-to-histopathology image translation using the pix2pix framework. Our method employs a conditional GAN architecture with multi-scale discriminators and a combined L1 + GAN loss, effectively alleviating deformation modeling difficulties inherent in conventional registration approaches. Ablation studies demonstrate that, compared to a U-Net baseline, our synthesized images exhibit fewer artifacts, achieving mutual information (MI) and structural similarity (SSIM) scores of 0.924 and 0.419, respectively. The implementation is publicly available, providing a reproducible, high-accuracy solution for spatial alignment in multi-omics integration.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Registration of histological and mass spectrometry imaging (MSI) allows for more precise identification of structural changes and chemical interactions in tissue. With histology and MSI having entirely different image formation processes and dimensionalities, registration of the two modalities remains an ongoing challenge. This work proposes a solution that synthesises histological images from MSI, using a pix2pix model, to effectively enable unimodal registration. Preliminary results show promising synthetic histology images with limited artifacts, achieving increases in mutual information (MI) and structural similarity index measures (SSIM) of +0.924 and +0.419, respectively, compared to a baseline U-Net model. Our source code is available on GitHub: https://github.com/kimberley/MIUA2025.

Problem

• Registering histology and MSI images is challenging due to differing dimensionalities.
• Synthesizing histology images from MSI enables unimodal registration via pix2pix.
• Proposed method improves mutual information and structural similarity metrics.

Innovation

• Uses pix2pix model for image synthesis
• Converts MSI to histological images
• Improves mutual information and SSIM