Problem
Research questions and friction points this paper is trying to address.
Implicit Generative Models
Mutual Information
Dependence Control
Singular Distributions
Density-Ratio Estimation
Controlling Dependence in Implicit Generative Models via Spread Mutual Information
🤖 AI Summary
This study addresses the challenges of evaluating mutual information in implicit generative models and the failure of gradient estimation caused by singular distributions. To overcome these issues, we propose Diffusion Mutual Information, a method that introduces a Gaussian diffusion kernel to smooth conditional and marginal densities. It integrates classifier-based log-density ratio learning with score-difference gradient estimation techniques. Furthermore, a weighted integral mechanism extends the constructed gradients back to the original singular distribution, effectively resolving the problem of insufficient density ratio overlap. Experimental results demonstrate that the proposed approach achieves reliable dependency control across various tasks, outperforming existing mutual information baselines.
📝 Abstract
Mutual information (MI) provides an objective for suppressing or encouraging statistical dependence in implicit generative models. However, direct MI evaluation is challenging in implicit models due to typically intractable densities. A remedy is estimating the generator gradient from the difference between conditional and marginal scores. This score difference can, in turn, be estimated by differentiating a log density ratio learned through classification. This construction nevertheless faces two difficulties: (i) singular distributions need not admit the required score functions, and (ii) poor overlap can hinder density-ratio estimation. We therefore introduce Spread Mutual Information (SMI), a weighted integral of MI across noise levels obtained by applying a common spreading kernel to the generated variable. Gaussian spreading yields smooth, strictly positive conditional and marginal densities, extending the gradient construction to distributions that may originally be singular. Across a variaty of experiments, SMI consistently achieves effective dependence control among MI-based methods and remains competitive with established task-specific approaches.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Spread Mutual Information
Implicit Generative Models
Score Matching
Density Ratio Estimation
Dependence Control
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
No related jobs found.
