Fisher-Orthogonal Projected Natural Gradient Descent for Continual Learning

2026-01-19

AI Summary

This work proposes the FOPNG optimizer to address catastrophic forgetting in continual learning, where neural networks tend to overwrite knowledge of previously learned tasks when acquiring new ones. By unifying natural gradient and orthogonal gradient concepts within an information-geometric framework, FOPNG projects the current gradient onto the orthogonal complement of prior task gradients under the Fisher information metric, thereby preserving previously acquired knowledge. The resulting update direction is invariant to parameterization and guarantees descent under the Fisher metric. An efficient implementation leveraging a diagonal approximation of the Fisher matrix enables FOPNG to achieve state-of-the-art performance across standard continual learning benchmarks, including Permuted-MNIST, Split-MNIST, Rotated-MNIST, Split-CIFAR10, and Split-CIFAR100.

Technology Category

• Natural Language Processing: Learning & Optimization for NLP
• Machine Learning: Life-Long and Continual Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Continual learning aims to enable neural networks to acquire new knowledge on sequential tasks. However, the key challenge in such settings is to learn new tasks without catastrophically forgetting previously learned tasks. We propose the Fisher-Orthogonal Projected Natural Gradient Descent (FOPNG) optimizer, which enforces Fisher-orthogonal constraints on parameter updates to preserve old task performance while learning new tasks. Unlike existing methods that operate in Euclidean parameter space, FOPNG projects gradients onto the Fisher-orthogonal complement of previous task gradients. This approach unifies natural gradient descent with orthogonal gradient methods within an information-geometric framework. We provide theoretical analysis deriving the projected update, describe efficient and practical implementations using the diagonal Fisher, and demonstrate strong results on standard continual learning benchmarks such as Permuted-MNIST, Split-MNIST, Rotated-MNIST, Split-CIFAR10, and Split-CIFAR100. Our code is available at https://github.com/ishirgarg/FOPNG.

Problem

• continual learning
• catastrophic forgetting
• task sequentiality
• neural networks
• knowledge retention

Innovation

• Fisher-orthogonal
• natural gradient descent
• continual learning
• catastrophic forgetting
• information geometry