Integrating Dual Prototypes for Task-Wise Adaption in Pre-Trained Model-Based Class-Incremental Learning

2024-11-26

arXiv.org

AI Summary

To mitigate catastrophic forgetting in pretrained models under class-incremental learning, this paper proposes a Dual-Prototype Adapter (DPA) framework. Methodologically: (1) it introduces task-level lightweight adapters to avoid full-parameter fine-tuning; (2) it establishes a dual-prototype mechanism—where the original prototype enables task-index inference for dynamic adapter selection, and the enhanced prototype improves discriminability among highly correlated classes; (3) it incorporates a center-adaptation loss to jointly optimize intra-class compactness and inter-class separability. The framework supports test-time adaptive adapter selection, achieving state-of-the-art performance across multiple benchmarks. It significantly alleviates forgetting while preserving strong transferability and incremental generalization capability.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Class-incremental learning (CIL) aims to acquire new classes while conserving historical knowledge incrementally. Despite existing pre-trained model (PTM) based methods performing excellently in CIL, it is better to fine-tune them on downstream incremental tasks with massive patterns unknown to PTMs. However, using task streams for fine-tuning could lead to catastrophic forgetting that will erase the knowledge in PTMs. This paper proposes the Dual Prototype network for Task-wise Adaption (DPTA) of PTM-based CIL. For each incremental learning task, a task-wise adapter module is built to fine-tune the PTM, where the center-adapt loss forces the representation to be more centrally clustered and class separable. The dual prototype network improves the prediction process by enabling test-time adapter selection, where the raw prototypes deduce several possible task indexes of test samples to select suitable adapter modules for PTM, and the augmented prototypes that could separate highly correlated classes are utilized to determine the final result. Experiments on several benchmark datasets demonstrate the state-of-the-art performance of DPTA. The code will be open-sourced after the paper is published.

Problem

• Addresses catastrophic forgetting in class-incremental learning with pre-trained models.
• Proposes a Dual Prototype network for task-wise adaptation in incremental tasks.
• Enhances class separability and representation clustering in incremental learning.

Innovation

• Dual Prototype network for task-wise adaption
• Task-wise adapter module for fine-tuning PTMs
• Test-time adapter selection for improved prediction