Fisher meets Feynman: score-based variational inference with a product of experts

2025-10-24

AI Summary

This paper addresses the challenge of modeling skewed, heavy-tailed, and multimodal target distributions in black-box variational inference (BBVI). To this end, it proposes an efficient PoE (Product of Experts)-based variational method. Methodologically, it introduces the Feynman identity from quantum field theory into BBVI to construct an analytically tractable PoE variational family; incorporates simplex-constrained latent variables and t-distributed experts; estimates expert weights via score matching; and optimizes via convex quadratic programming regularized by Fisher divergence, ensuring exponential convergence. Contributions include: (i) the first integration of the Feynman identity into BBVI for exact sampling from the variational family; (ii) a flexible, robust PoE architecture with provably convergent optimization; and (iii) empirically superior performance—demonstrating higher approximation accuracy, faster convergence, and greater expressive power than state-of-the-art baselines on both synthetic and real-world datasets.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We introduce a highly expressive yet distinctly tractable family for black-box variational inference (BBVI). Each member of this family is a weighted product of experts (PoE), and each weighted expert in the product is proportional to a multivariate $t$-distribution. These products of experts can model distributions with skew, heavy tails, and multiple modes, but to use them for BBVI, we must be able to sample from their densities. We show how to do this by reformulating these products of experts as latent variable models with auxiliary Dirichlet random variables. These Dirichlet variables emerge from a Feynman identity, originally developed for loop integrals in quantum field theory, that expresses the product of multiple fractions (or in our case, $t$-distributions) as an integral over the simplex. We leverage this simplicial latent space to draw weighted samples from these products of experts -- samples which BBVI then uses to find the PoE that best approximates a target density. Given a collection of experts, we derive an iterative procedure to optimize the exponents that determine their geometric weighting in the PoE. At each iteration, this procedure minimizes a regularized Fisher divergence to match the scores of the variational and target densities at a batch of samples drawn from the current approximation. This minimization reduces to a convex quadratic program, and we prove under general conditions that these updates converge exponentially fast to a near-optimal weighting of experts. We conclude by evaluating this approach on a variety of synthetic and real-world target distributions.

Problem

• Develops tractable product of experts for black-box variational inference
• Enables sampling from skewed/heavy-tailed distributions via latent variables
• Optimizes expert weighting through iterative Fisher divergence minimization

Innovation

• Product of experts with multivariate t-distributions
• Latent Dirichlet variables from Feynman identity
• Iterative optimization minimizing regularized Fisher divergence