Decoupled Multimodal Prototypes for Visual Recognition with Missing Modalities

2025-05-13

AI Summary

To address the significant performance degradation in multimodal visual recognition under modality missing conditions, this paper proposes a modality-decoupled prototype learning framework. Methodologically, we design modality-specific, missingness-aware class-level prototype heads that dynamically adapt prototypes to arbitrary modality combinations and missing rates. We further introduce missing-case-aware prompting, multimodal feature decoupling and alignment, and dynamic weighted fusion to enable fine-grained, scene-adaptive cross-modal reasoning. The framework is compatible with mainstream prompt-tuning paradigms. Extensive experiments demonstrate substantial robustness improvements across diverse modality-missing settings; notably, it achieves an average accuracy gain of 8.2% under high missing rates (>50%). Our contributions include: (1) a novel prototype learning architecture explicitly modeling modality missingness, (2) a unified decoupling–alignment–fusion mechanism for adaptive multimodal inference, and (3) state-of-the-art performance with strong generalizability to unseen missing patterns.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Multimodal learning enhances deep learning models by enabling them to perceive and understand information from multiple data modalities, such as visual and textual inputs. However, most existing approaches assume the availability of all modalities, an assumption that often fails in real-world applications. Recent works have introduced learnable missing-case-aware prompts to mitigate performance degradation caused by missing modalities while reducing the need for extensive model fine-tuning. Building upon the effectiveness of missing-case-aware handling for missing modalities, we propose a novel decoupled prototype-based output head, which leverages missing-case-aware class-wise prototypes tailored for each individual modality. This approach dynamically adapts to different missing modality scenarios and can be seamlessly integrated with existing prompt-based methods. Extensive experiments demonstrate that our proposed output head significantly improves performance across a wide range of missing-modality scenarios and varying missing rates.

Problem

• Handling missing modalities in multimodal learning systems
• Reducing performance degradation without extensive fine-tuning
• Adapting dynamically to various missing-modality scenarios

Innovation

• Decoupled prototype-based output head for missing modalities
• Dynamic adaptation to varying missing modality scenarios
• Seamless integration with existing prompt-based methods