🤖 AI Summary
This work investigates the precise conditions under which Transformers can rigorously implement Bayesian posterior inference. Framed within measure-theoretic and Markov kernel formalisms, the authors develop an abstract hierarchy extending from single-layer Bayesian Transformers to multi-layer stacked architectures. Without imposing additional structural assumptions, they formally prove that when the internal update mechanism satisfies a joint distribution condition, the forward computation of the Transformer is equivalent to exact Bayesian posterior updating. The central contribution lies in establishing, for the first time, a rigorous mathematical correspondence between the softmax attention mechanism and Bayesian inference. Specifically, the update kernel induced by the Transformer block is shown to coincide almost everywhere with the true posterior distribution, and the key-value mapping generated by attention constitutes a valid probability distribution.
📝 Abstract
We present a complete formal proof that transformer architectures, when their internal update mechanisms satisfy a Bayes joint-distribution condition, implement exact Bayesian posterior inference. Working within the measure-theoretic kernel framework, we define a hierarchy of abstractions -- from the core Bayesian transformer, through semantic transformers with explicit update kernels, to full transformer blocks with QKV/attention/residual/MLP pipelines, and finally multilayer stacks -- and prove at each level that the Bayes joint semantics implies the update kernel equals the posterior almost everywhere. For the block-level architecture, we derive the explicit Bayes formula through Radon-Nikodym differentiation and prove its normalization. We additionally prove that the softmax attention mechanism induces a valid probability distribution over keys, establishing the bridge between the abstract kernel framework and concrete attention implementations. The framework makes no architectural assumptions beyond the Markov kernel structure and exposes explicit conditions under which a transformer block is provably Bayesian. In essence, when this joint distribution condition is satisfied, the forward computation of a Transformer is formally equivalent to a rigorous Bayesian posterior update.