Stability of Measure-to-Measure Transformers on Sub-Gaussian Data

2026-10-06

AI Summary

This study addresses the limited theoretical foundations of Transformers and the unclear stability of their deep compositions. Adopting a measure-to-measure framework, it systematically investigates the mathematical properties of Transformers under sub-Gaussian data. Methodologically, the work integrates measure theory, optimal transport, and mean-field analysis to rigorously treat the cross-attention mechanism. Theoretically, it provides the first proof that this mapping preserves sub-Gaussianity and establishes its Hölder continuity with respect to the Wasserstein distance. Building upon these results, the project constructs a comprehensive theoretical framework for stability and finite-sample analysis, yielding error propagation estimates and approximation guarantees. Ultimately, this research lays a rigorous mathematical foundation for Transformer architectures.

Abstract

Transformers have exhibited impressive empirical success across various domains, but their theoretical foundations remain less developed. This work constitutes a mathematical study of the measure-to-measure operators defined by transformers. We show that transformers map sub-Gaussian inputs to sub-Gaussian outputs; this ensures that taking arbitrary-length compositions of the softmax operator is well-defined. We then show that transformers are Hölder continuous with respect to the 1-Wasserstein distance on appropriate spaces of sub-Gaussian inputs. This allows us to establish estimates on the error propagation along a transformer between a sub-Gaussian input and its empirical approximation. We also study a mean-field analog of the cross-attention mechanism, which is an operator from a pair of probability measures to a single probability measure. We show that cross-attention exhibits different Hölder regularity and sample-complexity in its two input arguments. Last, we apply our results to deduce approximation guarantees for measure-to-measure transformers. Together, these results provide a firm stability and finite-sample theory for transformers on sub-Gaussian data.

Problem

• Transformers
• stability
• sub-Gaussian data
• measure-to-measure operators
• finite-sample theory

Innovation

• Measure-to-Measure Transformers
• Sub-Gaussian data
• Wasserstein distance
• Hölder continuity
• Cross-attention