🤖 AI Summary
This work addresses the lack of a rigorous mathematical characterization of Transformers’ ability to model contextual relationships. The authors propose a measure-theoretic framework in which text is represented as a probability measure over a semantic embedding space, and inter-word contextual dependencies are modeled as coupling measures. They introduce the Sinkhorn Transformer architecture to approximate arbitrary continuous coupling functions. By integrating optimal transport theory with function approximation theory, the study establishes the first universal approximation theorem for Transformers in modeling contextual relations, proving that the proposed architecture can uniformly approximate any continuous coupling function. This result provides a formal theoretical guarantee for the semantic coupling mechanism and expressive power of Transformers.
📝 Abstract
Transformer architectures have achieved remarkable empirical success in modeling contextual relationships in natural language, yet a precise mathematical characterization of their expressive power remains incomplete. In this work, we introduce a measure-theoretic framework for contextual representations in which texts are modeled as probability measures over a semantic embedding space, and contextual relations between words, are represented as coupling measures between them. Within this setting, we introduce Sinkhorn Transformer, a transformer-like architecture. Our main result is a universal approximation theorem: any continuous coupling function between probability measures, that encodes the semantic relation coupling measure, can be uniformly approximated by a Sinkhorn Transformer with appropriate parameters.