ECA: Efficient Continual Alignment for Open-Ended Image-to-Text Generation

2026-06-10

AI Summary

This work addresses the challenge of continual learning and catastrophic forgetting in open-domain image-to-text generation, where temporal shifts in visually dominant categories disrupt model performance. To tackle this without replaying historical data, the authors propose a novel replay-free continual alignment mechanism that incrementally adapts the alignment module of a pretrained vision-language model. The approach dynamically expands the model architecture while preserving cross-modal representational capacity. Its core contributions include the introduction of the continual alignment paradigm, integrated with three key techniques: Mixture of Queries (MoQ), Fisher information–based dynamic expansion (FeDEx), and Dictionary Replay (DR). Evaluated on four newly curated real-world benchmarks, the method substantially mitigates forgetting and outperforms existing baselines.

Abstract

Incremental Learning (IL) for Open-ended Image-to-Text Generation (OpenITG) enables models to continuously generate accurate, contextually relevant text for new images while preserving previously acquired knowledge. Unlike prior studies, this paper addresses a more practical scenario in which the predominant category of visual data shifts over time as environments evolve. In this context, we introduce a new notion of continual alignment, which incrementally adapts the alignment module within pre-trained VLMs to preserve high-quality cross-modal representations. Based on this idea, we propose Efficient Continual Alignment (ECA), a novel exemplar-free IL approach for OpenITG. The key challenge is enabling the model to acquire new, task-specific features while minimizing interference with the established alignment without accessing raw data from previous tasks. To address this, ECA employs three core mechanisms: a Mixture of Query (MoQ) module that adapts task-specific query tokens, a Fisher Dynamic Expansion (FeDEx) that dynamically expands model structure based on a Fisher Information Matrix (FIM)-based metric, and an embedding dictionary with Dictionary Replay (DR) to retain past knowledge. To evaluate ECA's performance, we construct four new IL OpenITG benchmarks that better reflect real-world scenarios. Experimental results demonstrate that ECA significantly mitigates catastrophic forgetting and improves IL performance compared to baseline methods. Code and benchmarks are available at https://github.com/Snowball0823/ECA.

Problem

• Continual Learning
• Image-to-Text Generation
• Catastrophic Forgetting
• Cross-modal Alignment
• Open-ended Generation

Innovation

• Continual Alignment
• Exemplar-Free Incremental Learning
• Mixture of Query
• Fisher Dynamic Expansion
• Dictionary Replay