Physics Transformer: Tailoring Transformer for General PDE Prediction

2026-07-27

AI Summary

This work addresses the limitations of conventional Transformers in modeling the continuity and arbitrary discretization inherent in physical fields arising from partial differential equation (PDE) solutions. The authors propose the Physics Transformer, which introduces a novel tokenization mechanism based on function projection: physical fields are projected onto locally adaptive basis functions to yield tokens in a functional space. This approach preserves fine-grained spatial structure while enabling efficient global interactions and decoding at arbitrary locations. By integrating spatial-state factorized attention with continuous field reconstruction, the method establishes a new Transformer architecture that respects the functional nature of physical fields. The model achieves state-of-the-art performance across diverse benchmarks, including 2D PDE dynamics and industrial-scale 3D computational fluid dynamics, significantly improving accuracy in modeling fine-grained physical structures.

Abstract

Transformer architectures have attracted increasing attention for solving partial differential equations (PDEs), owing to their flexibility in handling irregular discretizations and their ability to capture long-range physical dependencies. However, unlike discrete language tokens or fixed-resolution image patches, observed physical fields are finite samples of underlying infinite-dimensional functions. Consequently, effectively applying Transformers to PDEs requires a tokenizer that respects the functional nature of physical fields and constructs physically expressive tokens from arbitrary discretizations.To this end, we propose \methodname{Physics Transformer}, a function-projection-based Transformer architecture for physical field prediction. Physics Transformer treats a physical field as a continuous function and partitions its discretization into locality-preserving spatial patches. Within each patch, it dynamically learns a set of adaptive local basis functions and projects the sampled field onto these bases to obtain compact physics tokens. The resulting tokens capture diverse latent physical states while preserving fine-scale spatial structures, enabling efficient global interaction through factorized attention across space and physical states. The projected representation further supports efficient decoding at arbitrary query locations. Extensive experiments on diverse benchmarks, ranging from two-dimensional PDE dynamics to industrial-scale three-dimensional CFD simulations, demonstrate that Physics Transformer accurately captures fine-grained physical structures and achieves state-of-the-art predictive performance. These results establish function projection as a practical and effective foundation for designing Transformer architectures for PDE solving.

Problem

• Transformer
• Partial Differential Equations
• Physics-Informed Learning
• Function Representation
• Tokenization

Innovation

• Physics Transformer
• function projection
• adaptive basis functions
• PDE prediction
• physics-informed tokens