CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning

2026-06-01

AI Summary

This work addresses the challenges of catastrophic forgetting and poor parameter efficiency in multimodal large language models under continual learning settings. The authors propose a task-isolation and adaptive expansion framework that decouples heterogeneous tasks through task-specific modules, dynamically instantiates necessary parameters via adaptive rank allocation, and employs a centroid-guided sparse routing mechanism to encourage expert reuse. Additionally, orthogonality constraints are introduced to prevent redundant learning of general-purpose capabilities. Evaluated across multiple multimodal continual learning benchmarks, the proposed method significantly outperforms existing approaches, effectively mitigating forgetting while achieving high parameter efficiency.

Abstract

Multimodal Large Language Models (MLLMs) unify heterogeneous vision-language tasks under a shared generative framework via instruction tuning, yet real-world deployment demands continuous capability expansion, making Multimodal Continual Instruction Tuning (MCIT) essential. Existing methods either update all tasks with a shared parameter set or allocate dedicated modules for each new task. Shared updates force heterogeneous tasks to compete, causing forgetting of learned capabilities. Conversely, isolated expansion prevents interference but severely limits parameter efficiency over long task streams. To address this dilemma, we propose CRAM. Specifically, by isolating task-specific patterns into independent modules, CRAM mitigates catastrophic forgetting across tasks. To further boost parameter efficiency, we utilize adaptive-rank instantiation to identify the capability gap between existing expert capability and new task demands, and dynamically allocate only the necessary parameters. To ensure stable reuse among tasks, centroid-guided routing recognizes and activates existing experts' capabilities, while an orthogonality penalty confines new updates to task-specific directions, preventing re-learning general capability. Extensive experiments across diverse benchmarks consistently demonstrate its superiority over existing methods.

Problem

• Multimodal Continual Instruction Tuning
• catastrophic forgetting
• parameter efficiency
• task interference
• capability expansion

Innovation

• Multimodal Continual Instruction Tuning
• Adaptive Mixture of Experts
• Centroid-Routing
• Parameter Efficiency
• Catastrophic Forgetting