Stability-Plasticity Balance via Singular-Vector Selection in LLM Continual Learning

2026-10-07

AI Summary

This study addresses the stability-plasticity dilemma in continual learning for large language models (LLMs), where mitigating catastrophic forgetting conflicts with adapting to new domains. We propose Singular Vector Channels (SVC), a method that establishes singular vector channels as natural units for managing this trade-off. By evaluating adaptation gains against forgetting costs, SVC adaptively selects trainable channels through Knee selection, Pareto front filtering, and Otsu thresholding, thereby allocating selective plasticity to optimize parameter efficiency. To our knowledge, this is the first work to leverage singular vector channels for such channel-wise plasticity management. Experiments across multiple LLM families and downstream tasks demonstrate that SVC significantly outperforms existing parameter-efficient fine-tuning baselines, effectively preserving pretrained knowledge while enhancing performance on novel domains.

Abstract

Domain-specific continual adaptation of LLMs risks catastrophic forgetting, creating a fundamental tension between acquiring new capabilities and preserving those learned during pretraining. PEFT mitigates this problem by restricting the number of trainable parameters, but existing methods lack a principled unit for deciding where plasticity should be allocated and stability should be preserved. We identify the singular-vector channel as a natural unit for managing this trade-off. Each channel represents an input-output transformation, which can be updated to acquire new knowledge or fixed to preserve pretrained capabilities. Based on this perspective, we introduce SVC, a parameter-efficient continual-learning method that selectively updates Singular-Vector Channels. Before fine-tuning, SVC uses domain-specific data to estimate each channel's adaptation benefit and a fixed public general-domain corpus only as a history activation proxy for estimating forgetting cost. It then adaptively selects trainable channels based on these scores via knee-based cost screening, Pareto-front filtering, and Otsu thresholding. Experimental results across four LLM families and eight downstream tasks show that SVC better preserves pretrained capabilities while achieving strong downstream performance relative to existing PEFT baselines. Further analysis of channel scoring and selection demonstrates that selective plasticity at the singular-vector-channel level enables effective continual LLM adaptation.

Problem

• continual learning
• catastrophic forgetting
• large language models
• stability-plasticity balance
• parameter-efficient fine-tuning

Innovation

• Continual Learning
• Singular-Vector Channels
• Parameter-Efficient Fine-Tuning
• Catastrophic Forgetting
• Stability-Plasticity Balance