🤖 AI Summary
This work addresses catastrophic forgetting in continual learning by proposing and systematically evaluating an improved implementation of Elastic Weight Consolidation (EWC). We reproduce EWC on the PermutedMNIST and RotatedMNIST benchmarks, augmenting the analysis with dropout integration and comprehensive hyperparameter sensitivity studies. Results demonstrate that EWC effectively balances knowledge retention and new-task adaptation: it substantially mitigates forgetting—outperforming unregularized SGD baselines—and enhances overall continual learning stability and generalization, albeit with a modest reduction in convergence speed on new tasks. Our study confirms EWC’s robustness across canonical non-stationary task sequences and reveals synergistic effects between regularization strength and architectural choices (e.g., dropout rate) on continual learning efficacy. Crucially, this work provides reproducible empirical evidence supporting lightweight, regularization-based continual learning approaches.
📝 Abstract
Catastrophic forgetting is the primary challenge that hinders continual learning, which refers to a neural network ability to sequentially learn multiple tasks while retaining previously acquired knowledge. Elastic Weight Consolidation, a regularization-based approach inspired by synaptic consolidation in biological neural systems, has been used to overcome this problem. In this study prior research is replicated and extended by evaluating EWC in supervised learning settings using the PermutedMNIST and RotatedMNIST benchmarks. Through systematic comparisons with L2 regularization and stochastic gradient descent (SGD) without regularization, we analyze how different approaches balance knowledge retention and adaptability. Our results confirm what was shown in previous research, showing that EWC significantly reduces forgetting compared to naive training while slightly compromising learning efficiency on new tasks. Moreover, we investigate the impact of dropout regularization and varying hyperparameters, offering insights into the generalization of EWC across diverse learning scenarios. These results underscore EWC's potential as a viable solution for lifelong learning in neural networks.