information theory

This work investigates the fundamental performance limits of learning and estimation tasks within an information-theoretic framework, independent of the computational capabilities of specific algorithms. By integrating tools from information theory and statistical learning theory—including metric entropy, VC dimension, Rademacher complexity, mutual information, and relative entropy—it systematically derives multiple upper bounds on generalization error. Simultaneously, leveraging Fano’s inequality together with covering and packing numbers, the study establishes information-theoretic lower bounds on minimax risk. The analysis unifies two complementary paradigms: one grounded in the geometric structure of metric spaces and the other based on information-theoretic measures. This synthesis yields a rigorous and broadly applicable theoretical framework for characterizing the optimal performance boundaries inherent to learning and estimation problems.

This work addresses the long-standing problem in partial information decomposition (PID) that synergistic information lacks a rigorous, computable definition. We propose a novel union information measure grounded in communication channel modeling—the first such channel-theoretic approach integrated into the PID framework. The measure satisfies the foundational axioms of minimality, monotonicity, and continuity, thereby yielding a unique, analytically tractable synergistic information quantity. Evaluated on canonical PID benchmark examples, our measure more accurately captures multivariate synergy than established alternatives such as (I_{min}) and (I_{ ext{proj}}), balancing theoretical soundness with computational feasibility. Beyond introducing the new measure, this work critically reexamines and systematically reconstructs the conceptual foundations of union and synergistic information representation in PID. The resulting framework provides an interpretable, reproducible analytical tool for synergy quantification, with direct applicability to neuroscience, complex systems analysis, and other domains requiring principled multivariate dependency assessment.

Modern information theory remains inaccessible to complex systems scientists due to its mathematical abstraction and lack of domain-specific interpretability. Method: This paper constructs an interpretable, cross-disciplinary information-theoretic framework grounded in Shannon entropy, mutual information, transfer entropy, and computational mechanics. It systematically integrates cutting-edge tools—including information dynamics, statistical complexity measures, partial information decomposition (PID), and effective network inference—with emphasis on physical interpretability and nonlinear dependency modeling. Contribution/Results: The framework provides the first unified exposition of how information theory characterizes system–environment coupling, part–whole architecture, and causal emergence in complex systems. By clarifying conceptual foundations and operationalizing abstract measures for empirical analysis, it substantially lowers the barrier to adoption. As a result, information theory is advanced as a general-purpose language for complexity modeling and rigorous causal analysis across disciplines.

The weak analogy between Shannon entropy and signed measures, coupled with the lack of geometric characterization for information sets, hinders a structural understanding of information. Method: We propose the Logarithmic Decomposition (LD) framework, which represents the information structure of random variables as definable “logarithmic atoms” over the sample space Ω. By extending Yeung’s I-measure, integrating signed measure theory with information geometry, and introducing the notion of logarithmic decomposability, the framework enables structured criteria for positive and negative entropy atoms. Contribution/Results: LD geometrically reconstructs common information and sufficient statistics; strictly distinguishes dyadic from triadic systems—beyond the capability of I-measure; unifies set-theoretic interpretations of mutual information, conditional entropy, and related quantities; establishes a foundation for quality-oriented information theory; and admits a natural extension to continuous distributions.

Classical information diagram theory has limited applicability due to its reliance on Shannon entropy and restrictive assumptions. Method: This paper proposes a generalized information decomposition framework axiomatized via monoid actions and the chain rule, unifying Hu’s theorem across diverse information measures—including Tsallis entropy, KL divergence, Kolmogorov complexity, and machine learning generalization error—for the first time. The approach integrates abstract algebra, probabilistic asymptotic analysis, and submodular function theory. Contributions/Results: (1) It establishes asymptotic equivalence between algorithmic information theory and classical information theory; (2) it proves that the expected interaction complexity converges to Shannon interaction information; and (3) it reveals asymptotic consistency between per-bit expected interaction complexity and information content for long sequences. By transcending Shannon-centric foundations, this framework substantially extends the theoretical scope and practical applicability of information diagrams in modeling complex systems.

Classical maximum entropy principle (MEP) relies on the system independence assumption in the Shore–Johnson axioms, which frequently fails in strongly correlated systems (e.g., economic or ecological networks), leading to systematic biases in Shannon-entropy-based inference. Method: We demonstrate that the Uffink–Jizba–Korbel (UJK) one-parameter generalized entropy family relaxes this assumption, offering a more robust entropy selection criterion for non-independent systems. By reformulating the Shore–Johnson axiomatization, we precisely delineate the domain of applicability for UJK entropies and establish a reproducible, transparent framework for entropy function selection and reporting. Contribution/Results: Empirical validation in economics (market interdependence modeling) and ecology (inference of species interactions) shows substantial improvements in distribution reconstruction accuracy and interpretability. This work is the first to systematically bridge foundational axiomatic principles with practical implementation guidelines, thereby advancing the reliable application of MEP in complex systems.

This paper addresses the fundamental quantity—excess growth rate—in portfolio theory, establishing its theoretical foundation from an information-theoretic perspective. Methodologically, it introduces three axiomatic characterization theorems that rigorously link the excess growth rate to relative entropy, the Jensen gap, and generalized Bregman divergences. By integrating tools from information theory, convex analysis, large deviations theory, and statistical physics, the work achieves cross-disciplinary modeling. The results reveal deep correspondences between the excess growth rate and Rényi entropy, cross-entropy, L. Campbell’s average coding length, the large deviation rate function, and the Helmholtz free energy in statistical physics. This study provides a unified, mathematically rigorous interpretation of the excess growth rate, solidifying its foundational role in quantitative finance. Most significantly, it establishes, for the first time, a systematic theoretical bridge between information theory and mathematical finance.

This work addresses the widespread misuse of information-theoretic measures in contemporary AI practice, which often stems from overlooking estimator assumptions, failure modes, and conditions required for reliable inference. The paper introduces the first unified decision framework that systematically integrates established measures—such as entropy and mutual information—with emerging ones like integrated information (Φ) and effective information. For each measure, the framework explicitly clarifies three core considerations: suitable AI application scenarios, appropriate estimators for given data types and dimensionalities, and common pitfalls leading to misinterpretation. Standardization and operationalization of measure selection are achieved through a combination of flowcharts, a primary decision table, and Bridge Box cognitive mapping techniques. The efficacy of this framework is empirically validated across three representative tasks: representation learning, temporal influence analysis, and agent complexity assessment.

This work addresses the lack of intuition in traditional derivations of exponential family distributions, which often obscure their information-theoretic and physical foundations in pedagogical contexts. By leveraging the principle of maximum entropy and requiring only elementary notions of entropy, the paper presents a concise and self-contained derivation that avoids complex constrained optimization. The core contribution demonstrates that, under constraints fixing the expected values of sufficient statistics, exponential family distributions uniquely maximize relative entropy with respect to a general base measure, and Shannon entropy in the special case of a uniform base measure. This approach reveals the fundamental connection between maximum entropy and exponential families from minimal assumptions, substantially streamlining the didactic exposition and fostering deeper integration of statistical theory with physical reasoning.

Traditional probability measures struggle to capture phase sensitivity and geometric structure between distributions, limiting the expressive power of information-theoretic metrics in complex systems. This work proposes a complex-valued probability measure framework that extends classical probability theory through phase modulation, establishing for the first time its rigorous mathematical foundation. Within this framework, three novel information measures—complex entropy, complex divergence, and complex metric—are developed to precisely characterize distributional uniformity, asymmetric dissimilarity, and symmetric distance, respectively. The approach reveals a formal analogy with Feynman’s path integral formulation and, by integrating tools from complex analysis, measure theory, and information theory, demonstrates superior geometric interpretability and empirical discriminative power over classical methods in two-sample nonparametric hypothesis testing.