entropy-based analysis

Shannon entropy relies on ergodic Markov processes, limiting its applicability to non-stationary and short-length sequences. Method: This paper proposes a unified combinatorial information-theoretic framework for arbitrary finite patterns. It introduces, for the first time, combinatorial definitions of information content and entropy compatible with classical information theory—free from assumptions of Markovity or stationarity—and develops a computable, normalized information estimator by integrating LZ77 compression, Kolmogorov complexity approximation, and asymptotic analysis. Contributions: (1) Enables rigorous, computable quantification of information and entropy in non-ergodic and small-sample settings; (2) The proposed entropy converges asymptotically to Shannon entropy; (3) Establishes universal comparability properties for information content, extending information measures to broader classes of discrete patterns.

Modern information theory remains inaccessible to complex systems scientists due to its mathematical abstraction and lack of domain-specific interpretability. Method: This paper constructs an interpretable, cross-disciplinary information-theoretic framework grounded in Shannon entropy, mutual information, transfer entropy, and computational mechanics. It systematically integrates cutting-edge tools—including information dynamics, statistical complexity measures, partial information decomposition (PID), and effective network inference—with emphasis on physical interpretability and nonlinear dependency modeling. Contribution/Results: The framework provides the first unified exposition of how information theory characterizes system–environment coupling, part–whole architecture, and causal emergence in complex systems. By clarifying conceptual foundations and operationalizing abstract measures for empirical analysis, it substantially lowers the barrier to adoption. As a result, information theory is advanced as a general-purpose language for complexity modeling and rigorous causal analysis across disciplines.

Machine learning practice has long lacked a unified, rigorous theoretical foundation; existing analyses rely heavily on empirical observations and fail to provide principled guidance for critical challenges such as generalization, robustness, and few-shot learning. Method: This paper establishes the first general learning theory framework unifying Bayesian statistics and Shannon information theory, characterizing the dynamic performance evolution of optimal Bayesian learners under streaming empirical data—rigorously accommodating i.i.d., sequential, hierarchical, and model-misspecified settings. Contribution/Results: The framework achieves the first cross-paradigm theoretical unification in learning theory, overcoming the degradation of classical asymptotic analysis with increasing data complexity. It yields tight, interpretable bounds on generalization error, data efficiency, and belief updating. These results provide computationally tractable, theoretically grounded foundations for distribution shift correction, few-shot learning, and robust algorithm design.

This paper addresses the challenge of quantifying uncertainty in Apparent Shannon Mutual Information (ASI)—a key metric for predictive quality assessment—whose estimation is hindered by the long-range dependencies, asymmetry, and heavy-tailed nature of the joint distribution $j(x,y)$, rendering classical mutual information inapplicable. We propose the first Bayesian ASI assessment framework: (1) We formally prove, for the first time, that ASI is the unique predictive quality measure satisfying both predictive and discriminative independence axioms; (2) We introduce a Dirichlet process mixture of skewed Student’s $t$-distributions, enabling the first Bayesian modeling of heavy-tailed, asymmetric $j(x,y)$ and principled uncertainty quantification of ASI; (3) We validate the framework on prostate cancer recurrence time prediction data, demonstrating its generality, robustness, and computational feasibility—particularly in real-world settings where $j(x,y)$ lacks closed-form expression.

The proliferation of entropy and complexity measures in nonlinear time-series analysis—many with overlapping functionalities—hampers method selection and sustainable software implementation. Method: We introduce EntropyHub, a high-performance, open-source Julia library unifying 1,638 entropy and complexity measures. Its core innovation is a mathematically rigorous, composable design—requiring only 2.3 lines of code per measure—built upon functional programming principles and a modular architecture deeply integrated with the DynamicalSystems.jl ecosystem. Contribution/Results: This design markedly enhances maintainability, extensibility, and cross-method fairness in benchmarking. Empirical evaluation demonstrates that EntropyHub surpasses existing tools across all key dimensions: scale of supported measures, computational performance, numerical robustness, and development efficiency. It has emerged as critical infrastructure for nonlinear dynamics research.

This work addresses the limitations of existing heuristic-based software testing metrics, which struggle to rigorously quantify a test suite’s ability to constrain the space of valid program implementations. For the first time, statistical mechanics is introduced into software testing: software entropy is formally defined, and a test suite is conceptualized as a macroscopic constraint over the program implementation space. By leveraging mutation analysis to approximate microscopic states, the authors estimate this entropy and propose an information-weighted metric for the distribution of test constraints. This novel measure reveals structural differences among test suites that traditional criteria—such as code coverage—fail to capture. Empirical validation on real-world projects demonstrates the approach’s effectiveness in reducing software entropy, while information weights quantify individual test cases’ contributions to constraining the program space.

To address the high data demand and low sampling efficiency in data-driven modeling of nonlinear dynamical systems, this paper introduces, for the first time, the Fisher Information Matrix (FIM) into the Sparse Identification of Nonlinear Dynamics (SINDy) framework. We propose an information-aware sampling strategy integrating FIM and Shannon entropy to enable discriminative selection of trajectory segments. Furthermore, we theoretically reveal how statistical bagging stabilizes the spectral properties of the FIM and establish a synergistic analytical paradigm unifying information theory and model discovery for quantifying data efficiency. Experiments across single-trajectory, parametrically controllable, and multi-initial-condition settings demonstrate substantial improvements: average prediction error decreases by 37%, information utilization increases by 2.1×, and high-accuracy sparse dynamical models are consistently recovered—both in chaotic and non-chaotic systems.

This work addresses the challenge of Shannon entropy estimation in realistic settings where the total number of species is unknown, a scenario in which traditional methods fail due to their reliance on a known species count. The authors propose the first application of the Pitman–Yor process to entropy estimation, formulating a Bayesian nonparametric model that treats the underlying distribution as an infinite-dimensional random measure. This approach enables stable and reliable diversity estimation even when observed species constitute only a small fraction of the true total. Theoretical analysis establishes the consistency of the proposed estimator under regularly varying distributions, while numerical experiments demonstrate its robustness and stability across diverse simulation settings. By explicitly accounting for the uncertainty inherent in species richness, the method significantly extends the applicability of entropy estimation to open-population scenarios.

Traditional probability measures struggle to capture phase sensitivity and geometric structure between distributions, limiting the expressive power of information-theoretic metrics in complex systems. This work proposes a complex-valued probability measure framework that extends classical probability theory through phase modulation, establishing for the first time its rigorous mathematical foundation. Within this framework, three novel information measures—complex entropy, complex divergence, and complex metric—are developed to precisely characterize distributional uniformity, asymmetric dissimilarity, and symmetric distance, respectively. The approach reveals a formal analogy with Feynman’s path integral formulation and, by integrating tools from complex analysis, measure theory, and information theory, demonstrates superior geometric interpretability and empirical discriminative power over classical methods in two-sample nonparametric hypothesis testing.

This work addresses the widespread misuse of information-theoretic measures in contemporary AI practice, which often stems from overlooking estimator assumptions, failure modes, and conditions required for reliable inference. The paper introduces the first unified decision framework that systematically integrates established measures—such as entropy and mutual information—with emerging ones like integrated information (Φ) and effective information. For each measure, the framework explicitly clarifies three core considerations: suitable AI application scenarios, appropriate estimators for given data types and dimensionalities, and common pitfalls leading to misinterpretation. Standardization and operationalization of measure selection are achieved through a combination of flowcharts, a primary decision table, and Bridge Box cognitive mapping techniques. The efficacy of this framework is empirically validated across three representative tasks: representation learning, temporal influence analysis, and agent complexity assessment.