REAL: Representation Enhanced Analytic Learning for Exemplar-free Class-incremental Learning

2024-03-20

arXiv.org

AI Summary

This work addresses the weak discriminative representation and severe catastrophic forgetting in exemplar-free class-incremental learning (EFCIL) caused by freezing the backbone network. Methodologically: (1) a supervised + self-supervised dual-stream pretraining strategy is proposed to enhance backbone representation robustness; (2) a knowledge distillation-guided representation enhancement mechanism is introduced to mitigate feature degradation under frozen backbone conditions; (3) analytical recursive least-squares optimization is adapted—first time—to the exemplar-free incremental setting, eliminating reliance on tunable backbones. Evaluated on CIFAR-100, ImageNet-100, and ImageNet-1k, the method significantly outperforms existing state-of-the-art exemplar-free approaches and achieves performance on par with or superior to exemplar-replay methods—despite requiring no stored samples. This establishes a new paradigm for efficient, low-overhead incremental learning.

Technology Category

• Machine Learning: Representation Learning
• Computer Vision: Representation Learning for Vision
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Representation, semantic annotation, enhancement, enrichments, access and/or integration of a variety of data on the Web
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Exemplar-free class-incremental learning (EFCIL) aims to mitigate catastrophic forgetting in class-incremental learning without available historical data. Compared with its counterpart (replay-based CIL) that stores historical samples, the EFCIL suffers more from forgetting issues under the exemplar-free constraint. In this paper, inspired by the recently developed analytic learning (AL) based CIL, we propose a representation enhanced analytic learning (REAL) for EFCIL. The REAL constructs a dual-stream base pretraining (DS-BPT) and a representation enhancing distillation (RED) process to enhance the representation of the extractor. The DS-BPT pretrains model in streams of both supervised learning and self-supervised contrastive learning (SSCL) for base knowledge extraction. The RED process distills the supervised knowledge to the SSCL pretrained backbone and facilitates a subsequent AL-basd CIL that converts the CIL to a recursive least-square problem. Our method addresses the issue of insufficient discriminability in representations of unseen data caused by a frozen backbone in the existing AL-based CIL. Empirical results on various datasets including CIFAR-100, ImageNet-100 and ImageNet-1k, demonstrate that our REAL outperforms the state-of-the-arts in EFCIL, and achieves comparable or even more superior performance compared with the replay-based methods.

Problem

• Enhance representation quality in exemplar-free class-incremental learning
• Improve backbone knowledge utilization for classifier training
• Mitigate catastrophic forgetting without historical training samples

Innovation

• Dual-stream pretraining enhances representation learning
• Representation distillation merges general and class-specific features
• Feature fusion buffer utilizes multi-layer backbone knowledge