Applications of Entropy in Data Analysis and Machine Learning: A Review

2024-12-01

Entropy

AI Summary

This paper addresses the fragmentation and lack of a unified theoretical foundation for entropy measures in data analysis and machine learning. We propose the first comprehensive, multi-source entropy theory taxonomy and generic framework tailored to data science. Grounded in the Shannon–Khinchin axioms, it unifies over ten entropy families—including Shannon, Rényi, Tsallis, Kolmogorov–Sinai, and von Neumann entropies—bridging perspectives from information theory, statistical learning, dynamical systems, and quantum probability. We systematically catalog over one hundred entropy-driven algorithms and empirically demonstrate that our framework significantly enhances robustness and interpretability in high-dimensional dimensionality reduction, time-series pattern recognition, and uncertainty quantification. Our core contributions are threefold: (1) a formal axiomatized generic representation of entropy; (2) a systematic methodology for entropy-based learning; and (3) a unified theoretical foundation for feature selection, clustering, anomaly detection, and model interpretation.

Technology Category

• Machine Learning: Information Theory
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Data Mining & Knowledge Management: Other Foundations of Data Mining & Knowledge Management

Application Category

• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Web Mining and Content Analysis: Machine learning and data science for the Web
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs

Abstract

Since its origin in the thermodynamics of the 19th century, the concept of entropy has also permeated other fields of physics and mathematics, such as Classical and Quantum Statistical Mechanics, Information Theory, Probability Theory, Ergodic Theory and the Theory of Dynamical Systems. Specifically, we are referring to the classical entropies: the Boltzmann–Gibbs, von Neumann, Shannon, Kolmogorov–Sinai and topological entropies. In addition to their common name, which is historically justified (as we briefly describe in this review), another commonality of the classical entropies is the important role that they have played and are still playing in the theory and applications of their respective fields and beyond. Therefore, it is not surprising that, in the course of time, many other instances of the overarching concept of entropy have been proposed, most of them tailored to specific purposes. Following the current usage, we will refer to all of them, whether classical or new, simply as entropies. In particular, the subject of this review is their applications in data analysis and machine learning. The reason for these particular applications is that entropies are very well suited to characterize probability mass distributions, typically generated by finite-state processes or symbolized signals. Therefore, we will focus on entropies defined as positive functionals on probability mass distributions and provide an axiomatic characterization that goes back to Shannon and Khinchin. Given the plethora of entropies in the literature, we have selected a representative group, including the classical ones. The applications summarized in this review nicely illustrate the power and versatility of entropy in data analysis and machine learning.

Problem

• Reviewing applications of entropy in data analysis and machine learning.
• Exploring classical and new entropies tailored for specific purposes.
• Characterizing probability distributions using axiomatic entropy definitions.

Innovation

• Entropy applications in data analysis
• Entropy tailored for machine learning
• Axiomatic characterization of entropy