Task-Oriented Rank Adaptation for Continual Learning in Text Classification

2026-10-01

AI Summary

This study addresses catastrophic forgetting and cross-task negative transfer in continual learning for text classification by proposing TORA, a geometric routing framework. By leveraging the structural similarity of LoRA low-rank adapters, TORA dynamically determines whether to share or isolate knowledge through a single geometric threshold, enabling efficient adapter routing and interference mitigation without requiring task identifiers. Evaluated across 15 benchmarks, TORA significantly enhances performance on compatible tasks and reduces training time while preserving accuracy on independent tasks. These results establish TORA as a promising new paradigm for continual learning under parameter-efficient fine-tuning.

Abstract

Continual learning (CL) in text classification faces two critical challenges: catastrophic forgetting and negative transfer across sequential tasks. Parameter-Efficient Fine-Tuning (PEFT) methods such as LoRA enable efficient adaptation by learning low-rank updates of the model parameters. However, these compact representations are normally trained in isolation, limiting their reuse across related tasks. We introduce Task-Oriented Rank Adaptation (TORA), a geometric routing framework that leverages the low-rank structure of LoRA adapters to decide whether to transfer knowledge from the most compatible expert (Boosting) or isolate the new task (Shielding) based on structural similarity. Evaluated across 15 diverse text classification benchmarks, TORA consistently avoids harmful routing decisions: compatible tasks exceed their isolated performance while reducing training time, and structurally distant tasks are protected from interference with no loss in accuracy. With a single geometric threshold and no reliance on task identities or predefined sequences, TORA provides a simple and effective approach for dynamic adapter routing in sequential text classification systems.

Problem

• Continual Learning
• Catastrophic Forgetting
• Negative Transfer
• Text Classification
• Parameter-Efficient Fine-Tuning

Innovation

• Continual Learning
• Task-Oriented Rank Adaptation
• LoRA
• Geometric Routing
• Parameter-Efficient Fine-Tuning