🤖 AI Summary
Shannon entropy relies on ergodic Markov processes, limiting its applicability to non-stationary and short-length sequences. Method: This paper proposes a unified combinatorial information-theoretic framework for arbitrary finite patterns. It introduces, for the first time, combinatorial definitions of information content and entropy compatible with classical information theory—free from assumptions of Markovity or stationarity—and develops a computable, normalized information estimator by integrating LZ77 compression, Kolmogorov complexity approximation, and asymptotic analysis. Contributions: (1) Enables rigorous, computable quantification of information and entropy in non-ergodic and small-sample settings; (2) The proposed entropy converges asymptotically to Shannon entropy; (3) Establishes universal comparability properties for information content, extending information measures to broader classes of discrete patterns.
📝 Abstract
A unified combinatorial definition of the information content and entropy of different types of patterns, compatible with the traditional concepts of information and entropy, going beyond the limitations of Shannon information interpretable for ergodic Markov processes. We compare the information content of various finite patterns and derive general properties of information quantity from these comparisons. Using these properties, we define normalized information estimation methods based on compression algorithms and Kolmogorov complexity. From a combinatorial point of view, we redefine the concept of entropy in a way that is asymptotically compatible with traditional entropy.