Turning Back Without Forgetting: Selective Backward Refinement for Parameter-Efficient Continual Learning

2026-05-31

AI Summary

This work addresses the limited backward knowledge transfer in prompt-based parameter-efficient continual learning, which arises from strict isolation among task-specific prompts. To overcome this challenge without relying on replay mechanisms, the authors propose SABER—a framework that selectively optimizes prompts of earlier tasks along non-interfering directions. Task relevance is determined by a novel criterion based on the geometric alignment of prompt gradients and the similarity of loss distributions across tasks, enabling safe and controlled bidirectional knowledge transfer. Experimental results demonstrate that SABER consistently improves average performance across multiple continual learning benchmarks and achieves robust bidirectional transfer when integrated with prominent large language models, including T5-Large, LLaMA, and Qwen.

Abstract

While prompt-based parameter-efficient continual learning mitigates catastrophic forgetting by isolating task-specific prompts, this isolation also limits later tasks from improving earlier ones, leaving backward knowledge transfer underexplored. We address this limitation by proposing Selective bAckward refinement for positive Backward knowledge transfER (SABER), a replay-free framework that enables controlled backward transfer in prompt-based continual learning. SABER determines when backward refinement is beneficial using complementary task-correlation criteria based on prompt-gradient geometry and loss-distribution similarity, and how to perform refinement safely by restricting updates to non-interfering directions in the prompt parameter space. Extensive experiments across multiple continual learning benchmarks and diverse pretrained backbones, including T5-Large, LLaMA, and Qwen, demonstrate that SABER consistently achieves positive backward transfer while maintaining strong overall average performance. Code is available at https://github.com/OptMN-Lab/SABER-ICML-2026/.

Problem

• continual learning
• catastrophic forgetting
• backward knowledge transfer
• prompt-based learning
• parameter-efficient learning

Innovation

• backward knowledge transfer
• parameter-efficient continual learning
• prompt-based learning
• task correlation
• non-interfering update