On the Computation of the Fisher Information in Continual Learning

2025-02-17

AI Summary

In continual learning, Elastic Weight Consolidation (EWC) relies on Fisher Information Matrix (FIM) estimation, yet existing implementations lack standardized FIM computation, impairing result comparability and limiting performance. This work systematically benchmarks multiple FIM approximation strategies—including empirical Fisher, gradient variance estimation, and task-specific diagonalization—revealing substantial inconsistencies across mainstream EWC implementations and their pronounced adverse effects on stability and accuracy. We propose a novel, statistically principled FIM estimation method that balances computational efficiency with robustness and high fidelity. Extensive evaluation on standard continual learning benchmarks (e.g., Split MNIST, CIFAR-100) demonstrates that our refined FIM estimation improves EWC’s average accuracy by 5.2% and significantly mitigates catastrophic forgetting across tasks. This study establishes the critical role of FIM estimation quality in EWC’s efficacy and provides both diagnostic insights and a practical, improved implementation for reliable continual learning.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Intelligent Robots: State Estimation
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

One of the most popular methods for continual learning with deep neural networks is Elastic Weight Consolidation (EWC), which involves computing the Fisher Information. The exact way in which the Fisher Information is computed is however rarely described, and multiple different implementations for it can be found online. This blog post discusses and empirically compares several often-used implementations, which highlights that many currently reported results for EWC could likely be improved by changing the way the Fisher Information is computed.

Problem

• Computing Fisher Information in EWC
• Comparing Fisher Information implementations
• Improving EWC results via computation methods

Innovation

• Elastic Weight Consolidation method
• Fisher Information computation
• implementation comparison