Natural Gradient Descent for Online Continual Learning

2026-03-21

AI Summary

This work addresses the challenges of catastrophic forgetting and slow convergence in online continual learning caused by non-i.i.d. data streams. It introduces, for the first time, natural gradient descent based on the KFAC approximation of the Fisher information matrix into this setting. By leveraging a more principled parameter update direction, the proposed method significantly enhances learning efficiency and stability across sequential tasks. Experimental results on standard benchmarks—including Split CIFAR-100, CORE50, and Split miniImageNet—demonstrate that the approach not only outperforms existing online continual learning algorithms but also consistently improves the performance of mainstream OCL techniques when integrated with them, thereby confirming its generality and effectiveness.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Online Continual Learning (OCL) for image classification represents a challenging subset of Continual Learning, focusing on classifying images from a stream without assuming data independence and identical distribution (i.i.d). The primary challenge in this context is to prevent catastrophic forgetting, where the model's performance on previous tasks deteriorates as it learns new ones. Although various strategies have been proposed to address this issue, achieving rapid convergence remains a significant challenge in the online setting. In this work, we introduce a novel approach to training OCL models that utilizes the Natural Gradient Descent optimizer, incorporating an approximation of the Fisher Information Matrix (FIM) through Kronecker Factored Approximate Curvature (KFAC). This method demonstrates substantial improvements in performance across all OCL methods, particularly when combined with existing OCL tricks, on datasets such as Split CIFAR-100, CORE50, and Split miniImageNet.

Problem

• Online Continual Learning
• Catastrophic Forgetting
• Non-i.i.d. Data
• Rapid Convergence

Innovation

• Natural Gradient Descent
• Online Continual Learning
• Fisher Information Matrix
• KFAC
• Catastrophic Forgetting