Beyond Classification: Dynamic Adapter Routing for Continual Multimodal Retrieval

2026-05-29

AI Summary

This work addresses the limitations of existing continual multimodal retrieval methods, which predominantly adopt class-incremental learning paradigms and struggle to handle dynamically shifting query–gallery distributions inherent in retrieval tasks. To this end, the paper introduces the first systematic evaluation benchmark tailored for continual multimodal retrieval and proposes a dynamic adapter routing mechanism that operates without relying on classification assumptions. The method leverages prototype-guided selection and fusion of task-specific adapters to enable effective knowledge retention and transfer. Extensive experiments demonstrate that the proposed approach significantly outperforms current baselines in both cross-domain continual retrieval and out-of-distribution generalization scenarios, confirming its efficacy and robustness.

Abstract

While retrieval is a core function of vision-language models, continually updating these models for retrieval tasks remains critically underexplored. Existing work often approaches continual retrieval through the lens of class-incremental learning (CIL), evaluating both standard CIL methods and retrieval-oriented adaptations in settings that may not fully capture the retrieval-specific dynamics. To address this, we introduce a new, principled evaluation framework for continual multimodal retrieval (CMR) spanning diverse visual domains, and systematically evaluate common approaches within this setting. Our empirical analysis shows that standard CIL methods fail to yield meaningful gains in our more challenging scenario. Therefore, we propose Dynamic Adapter Routing (DAR), a novel approach based on adapters selected through prototype-based routing and combined via model merging.DAR achieves superior performance over the previous baselines and demonstrates strong generalization under out-of-distribution evaluation. Our results highlights the unique challenges of CMR and encourages further research in this direction.

Problem

• continual multimodal retrieval
• vision-language models
• class-incremental learning
• retrieval dynamics
• out-of-distribution generalization

Innovation

• Dynamic Adapter Routing
• Continual Multimodal Retrieval
• Prototype-based Routing
• Model Merging
• Vision-Language Models