🤖 AI Summary
Bayesian inference under non-Gaussian likelihoods remains challenging due to the absence of a rigorous thermodynamic interpretation. Method: This work establishes a formal statistical-mechanical analogy for Bayesian inference, rigorously mapping Bayesian updating onto thermodynamic ensemble transformations. It systematically identifies partition functions, relative entropy, and thermodynamic quantities—work, heat, and free energy—with probabilistic update operations; introduces the concept of “likelihood work” and derives a Jarzynski-type equality; and defines an effective dimension based on the partition function as a novel model complexity measure. Contribution/Results: We derive an analytical expression for the path-wise relative entropy of sampling trajectories, enabling continuous-time Bayesian updating. The framework is validated on strongly non-Gaussian cosmological inverse problems, demonstrating superior modeling accuracy and computational robustness. This provides a unified thermodynamic interpretation of information-theoretic inference and delivers practical tools for statistical learning.
📝 Abstract
The significance of statistical physics concepts such as entropy extends far beyond classical thermodynamics. We interpret the similarity between partitions in statistical mechanics and partitions in Bayesian inference as an articulation of a result by Jaynes (1957), who clarified that thermodynamics is in essence a theory of information. In this, every sampling process has a mechanical analogue. Consequently, the divide between ensembles of samplers in parameter space and sampling from a mechanical system in thermodynamic equilibrium would be artificial. Based on this realisation, we construct a continuous modelling of a Bayes update akin to a transition between thermodynamic ensembles. This leads to an information theoretic interpretation of Jazinsky's equality, relating the expenditure of work to the influence of data via the likelihood. We propose one way to transfer the vocabulary and the formalism of thermodynamics (energy, work, heat) and statistical mechanics (partition functions) to statistical inference, starting from Bayes' law. Different kinds of inference processes are discussed and relative entropies are shown to follow from suitably constructed partitions as an analytical formulation of sampling processes. Lastly, we propose an effective dimension as a measure of system complexity. A numerical example from cosmology is put forward to illustrate these results.