Extended Information Geometry: Large Deviation Theory, Statistical Thermodynamics, and Empirical Counting Frequencies

2025-01-02

AI Summary

This paper establishes an axiomatic unification framework bridging information theory and statistical thermodynamics. Method: It introduces a novel synthesis of large deviations theory, Kolmogorov conditional expectation, and information projection, using empirical frequencies as the foundational driver to construct an extended information geometry; entropy functions—including Shannon entropy, mutual information, and relative entropy—are systematically derived and endowed with thermodynamic energy interpretations via Legendre–Fenchel parametrization of the empirical mean manifold. Contribution/Results: (1) It reveals thermodynamic-style additivity and intrinsic Riemannian geometric structure of entropies in the infinite-sample limit; (2) it unifies the additive properties of the empirical mean manifold with those of statistical thermodynamics; (3) it fundamentally extends information geometry from the space of probability distributions to the space of empirical frequencies, thereby providing a new paradigm for the geometric and physical interpretation of information.

Technology Category

• Machine Learning: Information Theory
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Natural Language Processing: Information Extraction

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Combinatorics, probabilities, and measurements are fundamental to understanding information. This work explores how the application of large deviation theory (LDT) in counting phenomena leads to the emergence of various entropy functions, including Shannon's entropy, mutual information, and relative and conditional entropies. In terms of these functions, we reveal an inherent geometrical structure through operations, including contractions, lift, change of basis, and projections. Legendre-Fenchel (LF) transform, which is central to both LDT and Gibbs' method of thermodynamics, offers a novel energetic description of data. The manifold of empirical mean values of statistical data ad infinitum has a parametrization using LF conjugates w.r.t. an entropy function; this gives rise to the additivity known in statistical thermodynamic energetics. This work extends current information geometry to information projection as defined through conditional expectations in Kolmogorov's probability theory.

Problem

• Geometric Structures of Information
• Statistical Thermodynamics
• Large Deviation Theory

Innovation

• Information Geometry
• Legendre-Fenchel Transform
• Information Projection