Advancing Multimodal Fusion on Heterogeneous Medical Data with Hybrid Geometry Attention

2026-07-21

AI Summary

Existing multimodal fusion methods in medical AI struggle to efficiently model complex cross-modal interactions, often suffering from high computational costs and limited generalization. To address these challenges, this work proposes CURE, a lightweight and scalable framework that progressively integrates heterogeneous modalities—such as imaging, clinical, and omics data—through cascaded unified representation learning and a novel HyFuse fusion layer. HyFuse synergistically combines multi-scale residual convolutions with a hybrid geometry-aware attention mechanism, enabling robust cross-modal relationship modeling while significantly reducing computational overhead. Notably, it supports arbitrary input modality ordering without performance degradation. Evaluated across 16 public datasets, CURE achieves up to a 3.97% average performance gain and reduces computational costs by as much as 87.8%, substantially outperforming current state-of-the-art approaches.

Abstract

Multimodal fusion learning (MFL) has shown great potential in the medical domain, where we are faced with disparate data modalities such as imaging, clinical records, and omics. However, existing MFL strategies face several major challenges. First, they struggle to capture complex cross-modal interactions effectively, which in turn limits performance improvements. Second, they incur high computational costs, restricting their applicability in resource-constrained healthcare AI applications. Finally, they are often designed and evaluated for narrow, fixed modality configurations (e.g., imaging-only, or specific pairs such as image and omics), which limits evidence of their adaptability and generalizability to broader collections of heterogeneous medical modalities. To address these challenges, we propose a novel MFL framework - Cascaded Unified Representation Learning for Efficient Fusion Network (CURE) - a lightweight and scalable framework that progressively integrates various modalities through a novel efficient Hybrid Geometry Aware Fusion layer (HyFuse), where each HyFuse layer is sequentially learned for each modality, making the framework adaptable and generalizable. Within HyFuse, an efficient residual convolution module captures rich multi-scale features to ensure cost-effective learning, while a hybrid-space aware attention mixer learns coarse-to-fine structural cues to better preserve cross-modal relationships. Complementary learnable late-fusion and shared information refinement modules are then employed to learn robust modality-order-invariant shared representations, which in turn yields consistent performance improvements. Extensive evaluations on 16 public datasets show that CURE outperforms leading multimodal fusion methods, boosting performance by up to 3.97% and lowering computational costs by up to 87.8%, ensuring more effective and reliable predictions.

Problem

• multimodal fusion
• heterogeneous medical data
• cross-modal interaction
• computational efficiency
• generalizability

Innovation

• Multimodal Fusion
• Hybrid Geometry Attention
• Efficient Learning
• Heterogeneous Medical Data
• Modality-Invariant Representation