Progressive Multimodal Alignment for Continual Instruction Tuning

2026-07-29

AI Summary

This work addresses projector forgetting in multimodal continual instruction tuning, which arises from visual distribution shifts and evolving instruction semantics. To mitigate this issue, the authors propose a progressive multimodal alignment framework that explicitly targets forgetting at the projector level. The approach preserves the pretrained projector as a stable anchor while dynamically expanding lightweight expert modules. It incorporates an extensible routing mechanism based on multimodal features and an on-demand activation strategy, achieving an effective balance between model stability and plasticity with only sublinear parameter growth. Experiments demonstrate that the proposed framework significantly outperforms existing methods on two MCIT benchmarks, generalizes across diverse multimodal large language models, and delivers robust, scalable continual learning performance.

Abstract

Multimodal Large Language Models (MLLMs) rely on a projector to align visual representations with the language embedding space, making it central to cross-modal understanding. In Multimodal Continual Instruction Tuning (MCIT), however, shifting visual distributions and evolving instruction semantics cause this shared projector to drift, leading to projector-level forgetting, an issue largely overlooked by methods that focus primarily on the LLM backbone. We introduce Progressive Multimodal Alignment (PMA), a framework that enables the projector to adapt continually while preserving previously learned alignment. PMA detects multimodal distribution shifts via a lightweight representation descriptor and progressively expands projector experts only when needed. An expandable router integrates expert outputs based on multimodal features, while the original pretrained projector is retained as a stable alignment anchor. This progressive mechanism balances stability and plasticity with sub-linear parameter growth and serves as a method-agnostic add-on to existing MCIT approaches. Extensive experiments on two recent MCIT benchmarks demonstrate that mitigating projector-level forgetting yields consistent gains over prior state-of-the-art methods when combined with PMA. Moreover, PMA scales across diverse MLLM backbones, demonstrating robust and broadly applicable MCIT performance.

Problem

• Multimodal Continual Instruction Tuning
• projector-level forgetting
• visual distribution shift
• instruction semantics evolution
• cross-modal alignment

Innovation

• Progressive Multimodal Alignment
• projector-level forgetting
• multimodal continual learning
• expandable projector experts
• stable alignment anchor