Continual Video-MLLM Adaptation over Evolving Domains

2026-07-21

AI Summary

This work addresses the challenges of catastrophic forgetting and cross-domain interference in continual learning for video multimodal large language models (Video-MLLMs) by proposing a Distribution-Aware Expert Routing framework (DAER). DAER freezes the pretrained Video-MLLM backbone and leverages lightweight, domain-isolated expert modules to preserve prior knowledge while adapting to new domains. The approach innovatively integrates intra-domain distribution-aware routing with inter-domain discriminative subspace routing, incorporating maximum mean discrepancy (MMD)-based prototype matching, parameter-efficient fine-tuning, task-id-free inference, and an adaptive domain merging mechanism. A two-stage optimization strategy further decouples general-purpose knowledge from domain-specific adaptation. Evaluated on a domain-incremental benchmark spanning ten VidQA datasets, DAER consistently outperforms existing methods across two prominent Video-MLLM backbones.

Abstract

Video multimodal large language models have shown strong capability in video understanding, yet their adaptation to sequentially evolving domains remains underexplored. In real-world deployments, video data often arrives continuously from heterogeneous domains, requiring the model to acquire new domain-specific knowledge without overwriting previously learned capabilities. Existing continual learning methods typically rely on shared adaptation spaces, which can induce severe cross-domain interference and catastrophic forgetting. We propose Distribution-Aware Expert Routing, a parameter-efficient framework for continual Video-MLLM adaptation over evolving domains. DAER maintains domain-isolated lightweight experts while keeping the pretrained Video-MLLM backbone frozen, thereby decoupling domain-specific adaptation from the general multimodal knowledge of the pretrained model. To enable fine-grained specialization, we introduce an intra-domain distribution-aware routing mechanism that matches each input to expert-level prototype reservoirs using MMD. To address the absence of task identities at inference time, we further propose an inter-domain routing mechanism that performs prototype matching in a discriminative subspace for robust domain identification. In addition, we introduce adaptive domain merging to improve parameter scalability and adopt a two-stage optimization strategy to stabilize expert specialization during continual learning. We evaluate DAER by curating a domain-incremental benchmark built from ten VidQA datasets covering diverse visual environments and reasoning demands. Experiments on two strong Video-MLLM backbones show that DAER consistently outperforms prior methods.

Problem

• continual learning
• video multimodal large language models
• catastrophic forgetting
• domain adaptation
• cross-domain interference

Innovation

• Continual Learning
• Video-MLLM
• Expert Routing
• Domain Adaptation
• Catastrophic Forgetting