Neural Scaling Laws of Transformer Operator Network

2026-09-27

AI Summary

This study addresses the lack of theoretical explanations for scaling laws in Transformer operator learning by constructing a framework that integrates Softmax attention mechanisms, Hölder regularity analysis, and intrinsic dimension exploration. Methodologically, it achieves discrete invariant outputs via a local-to-global principle while systematically analyzing approximation and generalization errors. The core contribution lies in rigorously deriving power-law generalization rates that surpass the logarithmic rates of conventional feedforward networks. Furthermore, this work verifies that convergence rates vary systematically with the intrinsic dimension of the input function class, thereby revealing the theoretical mechanisms underlying prediction error reduction as data and model scales increase.

Abstract

Transformers have emerged as powerful architectures for learning solution operators of physical systems. Empirically the prediction error has been observed to decrease when the data size and model size increase, suggesting neural scaling behavior. Yet a theoretical understanding of such scaling laws for transformer-based operator learning remains limited. In this work, we develop a theoretical framework for characterizing the approximation and generalization errors of transformer-based operator learning. Our analysis builds on a local-to-global approximation principle that is naturally aligned with the softmax attention mechanism and yields discretization-invariant output functions. On approximation theory, we derive a universal approximation error of transformer-based operator learning for H\"older-regular operators. On generalization theory, we establish a power scaling law between the prediction error and the training data size. The rate of convergence represented by the scaling exponent explicitly reflects the dimensions of the input and output domains, the regularity of the underlying functions and operators, and crucially, the intrinsic dimension of the input function class. By exploiting this intrinsic low-dimensional structure, our analysis yields a power-law generalization rate for operator learning, in contrast to the logarithmic-type power-law rates appearing in existing analyses of operator learning with feedforward neural networks. Numerical experiments validate the predicted power-law scaling and confirm that the convergence rate varies systematically with the intrinsic dimension of the input function class.

Problem

• Neural Scaling Laws
• Transformer Operator Network
• Operator Learning
• Approximation Theory
• Generalization Error

Innovation

• Transformer Operator Network
• Neural Scaling Laws
• Approximation Theory
• Generalization Error
• Intrinsic Dimension