Direct Message Approximation (DMA): A Consistency-Based Framework for Tractable Approximate Inference on Factor Graphs

2026-09-24

AI Summary

This study addresses the challenges of iterative scheduling, negative precision, and Dirac factor collapse inherent in Expectation Propagation (EP) and Variational Message Passing (VMP) within approximate message passing on factor graphs. To this end, we propose a direct message approximation framework that constructs messages via consistency conditions of normalizable factors, thereby eliminating inner loops and learning rate hyperparameters to enable efficient inference through a single forward-backward pass. Theoretically, we establish a main theorem bounding the KL divergence and provide the first O(1/r²) error guarantee for backward messages at product factors. Experimental results demonstrate that the proposed method effectively overcomes the negative precision issue while appropriately inflating predictive uncertainty in data-sparse regions.

Abstract

Approximate message passing on factor graphs underlies two dominant families of probabilistic inference algorithms: expectation propagation (EP) and variational message passing (VMP). Both methods approximate the marginal at each factor edge, forcing an iterative round-robin schedule, risking negative-precision messages, and, for VMP, collapsing to point estimates at Dirac-delta factors. We introduce Direct Message Approximation (DMA), which approximates factor-to-variable messages directly rather than the marginal. For normalisable factors, we define a consistency condition (requiring exactness when all other incoming messages are Dirac deltas) to guide message construction. We prove a master theorem (proper messages, any graph) bounding marginal KL from message KL, with three structural corollaries: Dirac-input consistency, no EP-style inner-loop iteration, and no negative-precision messages. Further, we prove a complementary $O(1/r^2)$ guarantee for the inherently improper backward message of the product factor, whose closed-form treatment has resisted prior work. As a concrete instantiation, we derive explicit DMA messages for the product and leaky-ReLU factors and assemble a Bayesian neural network (BNN) inference algorithm with one forward/backward sweep per training example and no gradient learning-rate hyperparameter, validating that the structural guarantees translate to predictive uncertainty that widens in data-sparse regions, including under model mismatch.

Problem

• approximate inference
• factor graphs
• message passing
• expectation propagation
• variational message passing

Innovation

• Direct Message Approximation
• Factor Graphs
• Consistency Condition
• Approximate Inference
• Bayesian Neural Networks