Technology Category
Machine Learning: Mixture of Experts (MoE)Cognitive Modeling & Cognitive Systems: Adaptive BehaviorNatural Language Processing: Learning & Optimization for NLP
F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learning
🤖 AI Summary
Parameter-efficient fine-tuning (PEFT) methods, particularly LoRA, underperform for large operator models in scientific machine learning due to theoretical limitations in Fourier layers. Method: This work introduces PEFT to scientific ML for the first time and proposes F-Adapter—a frequency-adaptive adapter architecture grounded in the spectral sparsity of physical systems. It allocates parameters dynamically across the frequency domain: high capacity at low frequencies and low capacity at high frequencies. Coupled with a spectral-complexity-driven module-width optimization strategy, F-Adapter jointly enhances parameter efficiency and approximation accuracy. Contribution/Results: On multiple 3D Navier–Stokes benchmarks, F-Adapter significantly outperforms LoRA and other state-of-the-art PEFT methods, achieving new SOTA performance while improving generalization and spectral fidelity.
Technology Category
Application Category
📝 Abstract
Parameter-efficient fine-tuning (PEFT) of powerful pre-trained models for complex downstream tasks has proven effective in vision and language processing, yet this paradigm remains unexplored in scientific machine learning, where the objective is to model complex physical systems. We conduct the first systematic study of PEFT for pre-trained Large Operator Models (LOMs) obtained by scaling variants of Fourier Neural Operator. First, we observe that the widely used Low-Rank Adaptation (LoRA) yields markedly poorer performance on LOMs than Adapter tuning. Then, we further theoretically establish that stacked LoRA incurs a depth-amplified lower bound on approximation error within Fourier layers, whereas adapters retain universal approximation capacity and, by concentrating parameters on energy-dominant low-frequency modes, attain exponentially decaying error with bottleneck width in the Fourier domain. Motivated by the robust empirical gains of adapters and by our theoretical characterization of PDE solutions as spectrally sparse, we introduce Frequency-Adaptive Adapter (F-Adapter). F-Adapter allocates adapter capacity based on spectral complexity, assigning higher-dimension modules to low-frequency components and lower-dimension modules to high-frequency components. Our F-Adapters establish state-of-the-art (SOTA) results on multiple challenging 3D Navier-Stokes benchmarks, markedly enhancing both generalization and spectral fidelity over LoRA and other PEFT techniques commonly used in LLMs. To the best of our knowledge, this work is the first to explore PEFT for scientific machine-learning and establishes F-Adapter as an effective paradigm for this domain.
Problem
Research questions and friction points this paper is trying to address.
Investigates parameter-efficient fine-tuning for scientific machine learning models
Addresses spectral approximation limitations in Fourier-based operator networks
Proposes frequency-adaptive capacity allocation for physical system modeling
Innovation
Methods, ideas, or system contributions that make the work stand out.
Frequency-adaptive adapter allocates capacity based on spectral complexity
Adapter parameters concentrate on energy-dominant low-frequency modes
Method achieves state-of-the-art results on 3D Navier-Stokes benchmarks
🔎 Similar Papers
No similar papers found.
