AdaHAT: Adaptive Hard Attention to the Task in Task-Incremental Learning

2026-08-02

AI Summary

This work addresses catastrophic forgetting in task-incremental learning, particularly under long task sequences where limited network capacity and an excessive proportion of static parameters hinder adaptation to new tasks. To tackle this challenge, the authors propose AdaHAT, an adaptive hard attention mechanism that extends Hard Attention to Tasks (HAT) by incorporating task importance evaluation and capacity-awareness. AdaHAT is the first architecture-level approach to conditionally activate and dynamically update critical parameters of previously learned tasks, thereby effectively balancing model stability and plasticity. Experimental results demonstrate that AdaHAT significantly outperforms existing methods across multiple benchmark datasets, achieving notably higher average accuracy in long task sequences while substantially mitigating forgetting and enhancing adaptability to new tasks.

Abstract

Catastrophic forgetting is a major problem in task-incremental learning, where neural networks tend to overwrite previously learned knowledge when trained on new tasks. A number of architecture-based approaches have been proposed to address this problem. However, the architecture-based approaches suffer from another problem related to network capacity when the networks learn long task sequences: As a network is trained on an increasing number of new tasks in a long task sequence, a growing proportion of active parameters becomes static to prevent forgetting of previously learned knowledge. In this paper, we propose Adaptive Hard Attention to the Task (AdaHAT) with an adaptive attention mechanism which allows adaptive updates to static parameters by taking into account the information about previous tasks on both the importance of these parameters to previous tasks and the current network capacity. Based on this idea, we develop a new neural network architecture incorporating our proposed AdaHAT mechanism. AdaHAT extends an existing architecture-based approach, Hard Attention to the Task (HAT), to better support task-incremental learning over long task sequences. We conduct experiments on a number of datasets and compare AdaHAT with task-incremental learning baselines including HAT. Our experimental results show that AdaHAT achieves better average performance across tasks than these baselines, especially on long task sequences, demonstrating the benefits from balancing the trade-off between stability and plasticity of a network when learning such sequences of tasks, alleviating the network capacity problem. Our code is available at pengxiang-wang.com/projects/continual-learning-arena.

Problem

• catastrophic forgetting
• task-incremental learning
• network capacity
• static parameters
• long task sequences

Innovation

• task-incremental learning
• catastrophic forgetting
• adaptive hard attention
• network capacity
• stability-plasticity trade-off