Functional Attention: From Pairwise Affinities to Functional Correspondences

2026-05-29

AI Summary

This work addresses a key limitation of existing Transformer-based operator learning methods, which discretize continuous fields into independent tokens and thereby neglect the global structure of function spaces, hindering their ability to model mappings between infinite-dimensional functions. To overcome this, the authors propose Functional Attention—a novel mechanism inspired by geometric functional maps—that generalizes attention from pointwise affinities to linear functional correspondences over function spaces. By replacing the conventional softmax with a structured linear operator and integrating adaptive basis construction, the method explicitly captures global dependencies. The resulting representation is compact, generalizable, and resolution-invariant, achieving state-of-the-art performance on tasks such as partial differential equation solving, 3D segmentation, and regression, while demonstrating strong robustness across diverse discretization schemes.

Abstract

Learning mappings between infinite-dimensional function spaces, or operator learning, is essential for many machine learning applications. Although transformer-based operators are popular, they often rely on token-wise attention. These methods treat continuous fields as discrete tokens and usually ignore the global functional structure. We introduce \emph{Functional Attention}, which reinterprets attention as a functional correspondence between adaptive bases. Inspired by geometric functional maps, our method replaces softmax affinities with structured linear operators. This yields a compact, generalizable, resolution-invariant representation that explicitly captures global dependencies. Experiments demonstrate that \emph{Functional Attention} can match state-of-the-art performance in many operator learning tasks, including solving PDEs, 3D segmentation, and regression, while remaining robust to varying discretizations. Project page is available at https://github.com/xjffff/FUNCATTN.

Problem

• operator learning
• functional correspondence
• attention mechanism
• function spaces
• global dependencies

Innovation

• Functional Attention
• Operator Learning
• Function Spaces
• Resolution Invariance
• Functional Correspondence