SLE-FNO: Single-Layer Extensions for Task-Agnostic Continual Learning in Fourier Neural Operators

2026-03-20

AI Summary

This work addresses the challenge of catastrophic forgetting in scientific machine learning when models encounter out-of-distribution data—such as changes in geometry, boundary conditions, or flow states—during continual learning. The authors propose SLE-FNO, a novel architecture that integrates a lightweight single-layer expansion (SLE) mechanism into the Fourier Neural Operator (FNO) for the first time. This approach enables task-agnostic, zero-forgetting continual learning without revisiting previous data and with only a minimal increase in model parameters. Evaluated on an image-to-image regression task involving 230 cerebral aneurysm CFD simulations, SLE-FNO substantially outperforms established continual learning methods—including EWC, LwF, replay, OGD, GEM, PiggyBack, and LoRA—achieving high predictive accuracy while effectively balancing model stability and plasticity.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Scientific machine learning is increasingly used to build surrogate models, yet most models are trained under a restrictive assumption in which future data follow the same distribution as the training set. In practice, new experimental conditions or simulation regimes may differ significantly, requiring extrapolation and model updates without re-access to prior data. This creates a need for continual learning (CL) frameworks that can adapt to distribution shifts while preventing catastrophic forgetting. Such challenges are pronounced in fluid dynamics, where changes in geometry, boundary conditions, or flow regimes induce non-trivial changes to the solution. Here, we introduce a new architecture-based approach (SLE-FNO) combining a Single-Layer Extension (SLE) with the Fourier Neural Operator (FNO) to support efficient CL. SLE-FNO was compared with a range of established CL methods, including Elastic Weight Consolidation (EWC), Learning without Forgetting (LwF), replay-based approaches, Orthogonal Gradient Descent (OGD), Gradient Episodic Memory (GEM), PiggyBack, and Low-Rank Approximation (LoRA), within an image-to-image regression setting. The models were trained to map transient concentration fields to time-averaged wall shear stress (TAWSS) in pulsatile aneurysmal blood flow. Tasks were derived from 230 computational fluid dynamics simulations grouped into four sequential and out-of-distribution configurations. Results show that replay-based methods and architecture-based approaches (PiggyBack, LoRA, and SLE-FNO) achieve the best retention, with SLE-FNO providing the strongest overall balance between plasticity and stability, achieving accuracy with zero forgetting and minimal additional parameters. Our findings highlight key differences between CL algorithms and introduce SLE-FNO as a promising strategy for adapting baseline models when extrapolation is required.

Problem

• continual learning
• catastrophic forgetting
• distribution shift
• scientific machine learning
• fluid dynamics

Innovation

• Continual Learning
• Fourier Neural Operator
• Single-Layer Extension
• Catastrophic Forgetting
• Task-Agnostic