SPACE-LoRA: Allocating Activation-Subspace Protection for Continual Learning

2026-09-28

AI Summary

This study addresses the problem of catastrophic forgetting in LoRA-based continual learning, where newly introduced adapters impose additive interference on the input directions of previously learned tasks. To mitigate this issue, we propose SPACE-LoRA, which introduces, for the first time, an activation subspace protection mechanism tailored to such interference. This approach suppresses the responses of new residual branches along task-sensitive directions associated with prior knowledge. Furthermore, leveraging the Fisher information matrix, SPACE-LoRA adaptively allocates protection coverage across individual modules. The primary contribution of this work lies in achieving module-level adaptive protection under a fixed rank constraint, thereby effectively alleviating forgetting and significantly enhancing both performance and stability in multi-task continual learning scenarios.

Abstract

This study addresses the catastrophic forgetting problem that occurs when sequentially learning successive tasks using Low-Rank Adaptation (LoRA) from a lifelong learning perspective. While existing approaches have primarily constrained parameter updates or learning subspaces to reduce interference with past knowledge, they have not fully considered additive interference. This occurs when a newly added residual adapter on top of a fixed past model generates non-zero responses along input directions important for old tasks, thereby altering previous predictions. To this end, we propose Subspace Protection with Allocated Capacity for Efficient Continual Adaptation (SPACE-LoRA). SPACE-LoRA directly suppresses the responses of the new residual branch along input activation directions that are important for old tasks and adaptively determines the protection coverage for each module based on past-task sensitivity estimated via a common Fisher sensitivity-based coverage target. Under a fixed LoRA rank, this approach adaptively adjusts module-specific protection coverage while suppressing interference along input directions sensitive to old tasks. We assess the effectiveness of activation-subspace protection in mitigating catastrophic forgetting and examine the role of sensitivity-guided protection in continual learning across diverse tasks. Code is available at https://anonymous.4open.science/r/SPACE-LoRA-7864.

Problem

• Catastrophic Forgetting
• Continual Learning
• Low-Rank Adaptation
• Additive Interference
• Lifelong Learning

Innovation

• Continual Learning
• Low-Rank Adaptation (LoRA)
• Catastrophic Forgetting
• Activation-Subspace Protection
• Fisher Sensitivity