NMINE: Normalized Mutual Information Neural Estimation

2026-07-30

AI Summary

Estimating mutual information between continuous high-dimensional variables remains challenging due to its unbounded nature and sensitivity to dimensionality, which compromises both accuracy and comparability; existing normalization methods further suffer from numerical instability in high dimensions. This work proposes the first fully neural normalized mutual information estimator, leveraging the Donsker–Varadhan representation within a MINE-like architecture to estimate mutual information. It introduces the MI-NEE framework, wherein neural networks learn the divergence between marginal distributions and a uniform reference distribution to stably recover entropy terms, thereby yielding a bounded and comparable normalized metric. Experiments on Gaussian data across dimensions 1 to 8 demonstrate that the proposed method significantly outperforms the KSG baseline, confirming its accuracy and robustness in estimating normalized dependence for high-dimensional continuous variables.

Abstract

Mutual information is a general measure of statistical dependence that captures both linear and nonlinear relationships between random variables. For continuous and multidimensional variables For continuous multidimensional variables, mutual information must be estimated from samples. Because mutual information is unbounded, its values are not directly comparable across datasets, dimensions, or applications. Normalized mutual information addresses this limitation by converting mutual information into a normalized dependency score. Recent work has demonstrated the practical value of normalized mutual information in applications such as molecular dynamics {arXiv:2405.04980} and interpretable machine learning {arXiv:2409.16768}, but existing estimators remain sensitive to dimensionality and numerical stability {arXiv:2410.07642}. In this paper, we propose a fully neural normalized mutual information estimator for continuous variables. The proposed approach combines a MINE-based neural mutual information estimator {arXiv:1801.04062} with MI-NEE-inspired neural marginal entropy estimators {arXiv:1905.12957}. Mutual information is estimated using the Donsker--Varadhan representation, while marginal entropies are estimated by learning the divergence between each marginal distribution and a uniform reference distribution, from which entropy is recovered. The resulting estimator provides a neural alternative to k-nearest-neighbor-based normalized mutual information estimation {arXiv:2405.04980}. Experiments on Gaussian data from one to eight dimensions show that the proposed estimator improves accuracy over a KSG-based normalized mutual information baseline. These results indicate that neural estimation is a promising direction for normalized dependency measurement in continuous multidimensional settings.

Problem

• normalized mutual information
• mutual information estimation
• continuous multidimensional variables
• numerical stability
• dimensionality sensitivity

Innovation

• Normalized Mutual Information
• Neural Estimation
• MINE
• Entropy Estimation
• Continuous Variables