Relative Parameter Importance in Task-Agnostic Replay-Free Continual Learning

2026-08-01

AI Summary

This work addresses the challenge of balancing stability and plasticity in replay-free, task-agnostic continual learning by introducing a novel measure of relative parameter importance. This metric dynamically evaluates each parameter’s relative contribution to current versus past tasks, enabling a differentiated regularization strategy. By incorporating the concept of relative importance for the first time, the method permits parameters with high historical but low relative importance to update more freely, thereby overcoming the rigidity of conventional protection mechanisms and facilitating both forward and backward knowledge transfer. The approach significantly outperforms existing methods on class-incremental and domain-incremental text classification benchmarks and offers a viable pathway for continual learning in text generation tasks.

Abstract

Achieving continual learning (CL) with deep neural networks requires balancing stability and plasticity while enabling knowledge transfer. In this work, we focus on offline learning algorithms under the constraints: (I) no access to training data from prior tasks (II) no access to task-id at inference time. We introduce a novel measure, the relative parameter-importance, which measures the relative importance of each parameter with respect to both the current and past tasks. Parameters with high relative importance are interpreted as more important for maintaining past-task stability and thus heavily regularised, whereas parameters with low relative-importance are allowed to be more freely updated. Unlike existing methods, our approach allows the update of parameters with high past-task importance when they have low relative-importance, thus enabling backward knowledge transfer in addition to tackling the stability-plasticity trade-off. We demonstrate improvements against state-of-the-art CL methods on both class-incremental and domain-incremental learning text classification problems and provide insights for extending our method to text generation problems. Code available at: https://github.com/itsmemala/LACL

Problem

• continual learning
• stability-plasticity trade-off
• task-agnostic
• replay-free
• knowledge transfer

Innovation

• relative parameter importance
• replay-free continual learning
• stability-plasticity trade-off
• backward knowledge transfer
• task-agnostic