CoDe-LoRA: Mitigating the Orthogonality Dilemma in Continual Learning of LLMs via Knowledge Consolidation and Decoupling

2026-10-06

AI Summary

This study addresses the "orthogonality dilemma" in continual learning for large language models, where strict orthogonality constraints impede knowledge sharing and induce catastrophic forgetting. To overcome this, we propose CoDe-LoRA, a method that decouples general and task-specific knowledge through adaptive null-space projection and semantic routing mechanisms. This design enables effective transfer and accumulation of cross-task representations without requiring experience replay. Extensive experiments demonstrate that CoDe-LoRA achieves state-of-the-art average accuracy across four backbone architectures and three benchmark datasets, significantly outperforming existing approaches.

Abstract

Continual learning (CL) is essential for Large Language Models (LLMs) to sequentially adapt to evolving tasks. To mitigate catastrophic forgetting, recent advances implement low-rank adaptation with orthogonal projections (e.g., O-LoRA) to isolate task parameters. However, we reveal that such strict geometric constraints trigger an "Orthogonality Dilemma": rigid parameter isolation impedes the transfer and accumulation of shared representations across semantically related tasks. In this work, we propose a new replay-free method, called Consolidation and Decoupling LoRA (CoDe-LoRA), for CL of LLMs. CoDe-LoRA disentangles the learning process into Consolidating Universal Knowledge and Decoupling Task-Specific Knowledge. To achieve this, CoDe-LoRA leverages an adaptive null space projection mechanism and semantic routing to balance knowledge accumulation with task-specific adaptation. Experimental results across four backbones and three CL benchmarks show that CoDe-LoRA achieves the best average accuracy. Our code is available at https://github.com/Estrellajer/CoDe-LoRA.

Problem

• Continual Learning
• Large Language Models
• Catastrophic Forgetting
• Orthogonality Dilemma
• Low-Rank Adaptation

Innovation

• Continual Learning
• Low-Rank Adaptation
• Orthogonality Dilemma
• Null Space Projection
• Semantic Routing