LoRA-Based Continual Learning with Constraints on Critical Parameter Changes

2025-04-18

AI Summary

To address catastrophic forgetting in LoRA-based continual learning—where orthogonal fine-tuning still induces drift in critical pre-trained parameters—we propose a stability-plasticity-balanced approach. Specifically, we introduce the first explicit freezing of task-critical parameter matrices from prior tasks within Vision Transformers (ViTs), coupled with a novel QR-decomposition-based LoRA composition paradigm (LoRAC) to enhance representational capacity and task adaptability of orthogonal LoRA. Our method integrates parameter importance estimation, orthogonal constraint optimization, and LoRA fine-tuning. Evaluated on Split CIFAR-100, it achieves a 6.35% absolute accuracy gain and reduces average forgetting by 3.24%, establishing new state-of-the-art performance in continual learning.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Life-Long and Continual Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

LoRA-based continual learning represents a promising avenue for leveraging pre-trained models in downstream continual learning tasks. Recent studies have shown that orthogonal LoRA tuning effectively mitigates forgetting. However, this work unveils that under orthogonal LoRA tuning, the critical parameters for pre-tasks still change notably after learning post-tasks. To address this problem, we directly propose freezing the most critical parameter matrices in the Vision Transformer (ViT) for pre-tasks before learning post-tasks. In addition, building on orthogonal LoRA tuning, we propose orthogonal LoRA composition (LoRAC) based on QR decomposition, which may further enhance the plasticity of our method. Elaborate ablation studies and extensive comparisons demonstrate the effectiveness of our proposed method. Our results indicate that our method achieves state-of-the-art (SOTA) performance on several well-known continual learning benchmarks. For instance, on the Split CIFAR-100 dataset, our method shows a 6.35% improvement in accuracy and a 3.24% reduction in forgetting compared to previous methods. Our code is available at https://github.com/learninginvision/LoRAC-IPC.

Problem

• Mitigates forgetting in continual learning tasks
• Freezes critical ViT parameters for pre-tasks
• Enhances plasticity via orthogonal LoRA composition

Innovation

• Freezing critical ViT parameter matrices
• Orthogonal LoRA composition (LoRAC) via QR
• Enhancing plasticity in continual learning