Cluster Attention Neural Operators for Solving Parametric Partial Differential Equations

2026-09-30

AI Summary

This study addresses the computational redundancy of traditional PDE simulations, the high complexity of Transformer attention, and the information loss inherent in spatial projections by proposing the CANO model. Its core innovation lies in introducing dynamic query clustering under a cross-attention mechanism while preserving full-resolution key-value mappings. This design circumvents the losses associated with patch-based compression and the constraints of weight sharing, thereby enabling efficient, lossless capture of global dependencies. Experimental evaluations demonstrate that CANO achieves state-of-the-art accuracy across fluid dynamics, solid mechanics, and irregular geometry benchmarks, exhibiting superior geometric adaptability and temporal consistency.

Abstract

Traditional simulations of parametric partial differential equations (PDEs) rely on repetitive computations for each parameter, which makes high-fidelity design impractical. Neural operators address this issue by learning solution operators, accelerating parameter-space mapping by orders of magnitude. Recent Transformer-based neural operators attempt to capture global dependencies, but often at the cost of quadratic attention complexity. Transolver resolves this problem by projecting physical states into a reduced slice space for attention computation. Although fast, this projection sacrifices fine spatial information. Moreover, by operating in this reduced space with shared weights across attention heads, it may constrain the model's flexibility, thereby limiting its capacity to capture complex phenomena. To address these issues, we propose the Cluster Attention Neural Operator (CANO), which reformulates attention via a novel cross-attention mechanism that dynamically clusters queries while preserving full-resolution keys and values. This avoids slice compression loss and removes weight-sharing limits. At the same time, the model remains fast without losing global interactions. Empirically, CANO achieves state-of-the-art performance across canonical PDE benchmarks, covering fluid and solid dynamics (e.g., Navier-Stokes, Airfoil, Plasticity), irregular unstructured geometries (e.g., Pipe Turbulence, Composites), and long-term temporal rollouts. Across solid deformation and turbulent flow benchmarks, CANO achieves lower errors than baselines and exhibits strong geometric adaptability and temporal consistency.

Problem

• Parametric Partial Differential Equations
• Neural Operators
• Attention Complexity
• Spatial Information Loss
• Weight Sharing

Innovation

• Neural Operator
• Cluster Attention
• Cross-Attention Mechanism
• Parametric PDEs
• Transformer