Universal Approximation of Measure-to-Measure Operators by Pushforwards

2026-09-28

AI Summary

This study addresses the limitation that universal approximation of distribution-to-distribution mapping operators readily fails under atomic inputs, exposing deficiencies in existing theories. To overcome this, we propose the uniform level set condition, integrating measure theory, optimal transport theory, and Transformer architecture analysis to construct a continuous measure-dependent pushforward model based on the Wasserstein distance. We prove that, on compact sets satisfying this condition, the pushforward model can uniformly approximate arbitrary continuous operators, thereby effectively overcoming the constraints imposed by atomic inputs. This work establishes a comprehensive theoretical framework for universal approximation, providing a rigorous foundation for measure-theoretic Transformers and cross-attention mechanisms.

Abstract

Many learning tasks map an input distribution to an output distribution. A natural way to model such an operator is to transform each input sample using a continuous function that may depend on the entire input distribution, and then take the distribution of the transformed samples. This defines a measure-dependent pushforward model and includes measure-theoretic formulations of transformers. We ask when such models can approximate arbitrary continuous operators between spaces of probability measures. We first show that universal approximation fails when atomic inputs are allowed: some continuous measure-to-measure operators that split or redistribute atomic mass cannot be approximated arbitrarily well by deterministic pushforward models. We then introduce the uniform level set condition, which requires a continuous measure-dependent scalarization whose shrinking level set neighborhoods carry uniformly vanishing mass over the input family. This condition is satisfied, in particular, by compact families of absolutely continuous measures. On every compact family satisfying this condition, we prove that any continuous measure-to-measure operator with outputs of finite $p$-th moment can be uniformly approximated, in the $p$-Wasserstein distance, by continuous measure-dependent pushforwards. Combining our theorem with existing approximation results for measure-dependent in-context maps yields universal approximation by measure-theoretic transformers. We also extend the framework to continuously-varying source measures, yielding a corresponding universality result for a class of pushforward models that are closely aligned with cross-attention architectures.

Problem

• universal approximation
• measure-to-measure operators
• pushforward models
• probability measures
• transformers

Innovation

• Measure-dependent pushforward
• Universal approximation
• Uniform level set condition
• Wasserstein distance
• Measure-theoretic transformers