A 3D Cross-modal Keypoint Descriptor for MR-US Matching and Registration

2025-07-24

AI Summary

To address the challenging registration between preoperative MRI and intraoperative real-time ultrasound (iUS) caused by substantial modality discrepancy, this paper proposes a novel 3D cross-modal keypoint descriptor. Methodologically, we synthesize patient-specific ultrasound images from MRI to construct a supervised contrastive learning framework; it integrates probabilistic keypoint detection, curriculum-based triplet loss, and dynamic hard negative mining to learn a robust, shared descriptor space. Our contribution is the first synthetic-data-driven, initialization-free, interpretable, and rotation-invariant cross-modal matching framework. Evaluated on the ReMIND dataset, our method achieves a mean matching accuracy of 69.8% and a target registration error of only 2.39 mm—significantly outperforming current state-of-the-art approaches.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Intraoperative registration of real-time ultrasound (iUS) to preoperative Magnetic Resonance Imaging (MRI) remains an unsolved problem due to severe modality-specific differences in appearance, resolution, and field-of-view. To address this, we propose a novel 3D cross-modal keypoint descriptor for MRI-iUS matching and registration. Our approach employs a patient-specific matching-by-synthesis approach, generating synthetic iUS volumes from preoperative MRI. This enables supervised contrastive training to learn a shared descriptor space. A probabilistic keypoint detection strategy is then employed to identify anatomically salient and modality-consistent locations. During training, a curriculum-based triplet loss with dynamic hard negative mining is used to learn descriptors that are i) robust to iUS artifacts such as speckle noise and limited coverage, and ii) rotation-invariant . At inference, the method detects keypoints in MR and real iUS images and identifies sparse matches, which are then used to perform rigid registration. Our approach is evaluated using 3D MRI-iUS pairs from the ReMIND dataset. Experiments show that our approach outperforms state-of-the-art keypoint matching methods across 11 patients, with an average precision of $69.8%$. For image registration, our method achieves a competitive mean Target Registration Error of 2.39 mm on the ReMIND2Reg benchmark. Compared to existing iUS-MR registration approach, our framework is interpretable, requires no manual initialization, and shows robustness to iUS field-of-view variation. Code is available at https://github.com/morozovdd/CrossKEY.

Problem

• Intraoperative MR-US registration faces modality-specific differences
• Proposing 3D cross-modal keypoint descriptor for MRI-US matching
• Achieving robust, interpretable registration without manual initialization

Innovation

• 3D cross-modal keypoint descriptor for MRI-iUS matching
• Patient-specific synthetic iUS volumes generation
• Curriculum-based triplet loss with dynamic mining