AlphaWiSE: Adaptive Weight Interpolation for Continual Multimodal Representation Learning

2026-07-16

AI Summary

This work addresses the challenge in continual multimodal learning where models, when sequentially acquiring new tasks, tend to disrupt previously learned cross-modal alignment. To mitigate this issue, the authors propose AlphaWiSE—a post-hoc weight interpolation method that introduces parameter space interpolation into continual multimodal learning for the first time. AlphaWiSE adaptively learns a shared scalar coefficient for each aligned parameter tensor between two frozen source checkpoints, leveraging a small replay buffer and a cross-modal embedding preservation mechanism. Notably, it achieves a flexible trade-off between stability and plasticity without altering model architecture or incurring additional inference overhead. Experimental results demonstrate that AlphaWiSE significantly outperforms existing continual learning baselines on audio–image–text multimodal retrieval tasks.

Abstract

Multimodal models such as CLIP learn a shared embedding space for cross-modal retrieval, but continual adaptation to sequentially arriving data can disrupt the cross-modal alignment acquired from earlier phases. Conventional continual-learning methods return a single checkpoint, which commits every retrieval direction to the same stability-plasticity trade-off. We propose AlphaWiSE, a post-hoc weight-space interpolation method that composes two frozen source checkpoints. For each aligned parameter tensor identified by its checkpoint key, AlphaWiSE fits one scalar interpolation coefficient shared by all tensor entries. The coefficients are fitted on a smaller exemplar memory and used to materialize one interpolated checkpoint. The deployed model has the same architecture and parameter count as either source checkpoint, which does not require additional inference time. Extensive experiments on audio-image-text retrieval show consistent improvements over strong continual-learning baselines across multiple retrieval directions and evaluation metrics.

Problem

• continual learning
• multimodal representation learning
• cross-modal alignment
• stability-plasticity trade-off
• retrieval

Innovation

• weight interpolation
• continual learning
• multimodal representation
• post-hoc adaptation
• cross-modal alignment