MMG: Mutual Information Estimation via the MMSE Gap in Diffusion

2025-09-24

AI Summary

This work addresses the challenge of accurately estimating mutual information (MI) in high-dimensional complex systems. We propose a novel method grounded in denoising diffusion models, establishing a rigorous information-theoretic connection between diffusion processes and MI. Specifically, we derive an exact identity: MI = ½∫MMSE_gap(dsnr), where the integrand is the mean squared error (MSE) gap between conditional and unconditional denoising tasks across the full signal-to-noise ratio (SNR) spectrum. Our estimator jointly leverages conditional and unconditional score function estimation alongside adaptive importance sampling, substantially improving stability and scalability—particularly in high-MI regimes. Experiments demonstrate that our approach outperforms existing classical and score-based diffusion MI estimators under self-consistency evaluation and maintains robustness even under strong statistical dependencies.

Technology Category

• Machine Learning: Information Theory
• Computer Vision: Diffusion Models for Vision
• Intelligent Robots: State Estimation

Application Category

• Web Mining and Content Analysis: Content-based information diffusion
• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Mutual information (MI) is one of the most general ways to measure relationships between random variables, but estimating this quantity for complex systems is challenging. Denoising diffusion models have recently set a new bar for density estimation, so it is natural to consider whether these methods could also be used to improve MI estimation. Using the recently introduced information-theoretic formulation of denoising diffusion models, we show the diffusion models can be used in a straightforward way to estimate MI. In particular, the MI corresponds to half the gap in the Minimum Mean Square Error (MMSE) between conditional and unconditional diffusion, integrated over all Signal-to-Noise-Ratios (SNRs) in the noising process. Our approach not only passes self-consistency tests but also outperforms traditional and score-based diffusion MI estimators. Furthermore, our method leverages adaptive importance sampling to achieve scalable MI estimation, while maintaining strong performance even when the MI is high.

Problem

• Estimating mutual information for complex systems is challenging
• Using diffusion models to improve mutual information estimation accuracy
• Achieving scalable MI estimation while maintaining high performance

Innovation

• Uses diffusion models for mutual information estimation
• Leverages MMSE gap between conditional and unconditional diffusion
• Applies adaptive importance sampling for scalable estimation