TailLoR: Protecting Principal Components in Parameter-Efficient Continual Learning

2026-06-04

AI Summary

This work addresses catastrophic forgetting in parameter-efficient continual learning by proposing TailLoR, a method that constructs a fixed reference frame based on the singular vectors of pre-trained weights and applies low-rank updates to the singular value matrix. TailLoR introduces a soft spectral penalty to steer adaptation away from dominant singular directions, thereby channeling parameter updates toward the long-tail spectral coordinates. This approach is the first to explicitly preserve principal components within a spectral decomposition framework while leveraging the long-tail spectrum for flexible adaptation, effectively balancing model stability and plasticity. Experimental results demonstrate that TailLoR substantially reduces interference across tasks, significantly improves continual learning performance, and achieves these gains with an extremely small number of trainable parameters.

Abstract

Parameter-efficient finetuning methods based on spectral decomposition have enabled progress in Continual Learning. In this paper we introduce TailLoR, which utilizes the singular bases U and V of the pre-trained weights as a fixed reference frame to learn a low-rank update applied to the singular value matrix. A soft spectral penalty discourages updates aligned with dominant singular directions, reducing interference while routing fine-grained adaptation into the highly flexible, long-tail spectral coordinates.

Problem

• Continual Learning
• Parameter-Efficient Finetuning
• Spectral Decomposition
• Principal Components
• Low-Rank Update

Innovation

• TailLoR
• spectral decomposition
• low-rank adaptation
• continual learning
• singular value matrix