Variational inference for hierarchical models with conditional scale and skewness corrections

2025-03-23

AI Summary

Gaussian variational approximation, while computationally efficient, fails to capture complex posterior structures—particularly skewness—especially in hierarchical Bayesian models. To address this, we propose a conditional skewness-corrected variational inference method: skewness correction is explicitly embedded into the variational family definition, enabling the first adaptive coupling of skewness parameters to local conditional posteriors. By decoupling location, scale, and skewness modeling, our approach avoids introducing extraneous global skewness parameters and supports joint optimization of both global and local parameters. Built upon a hierarchical variational family, the method incurs only marginal additional computational cost over standard Gaussian variational inference. Experiments on generalized linear mixed models and multinomial logit discrete choice models demonstrate substantial improvements in posterior approximation accuracy, effectively mitigating systematic bias arising from skewness neglect in conventional methods.

Technology Category

• Machine Learning: Bayesian Learning
• Reasoning under Uncertainty: Relational Probabilistic Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Gaussian variational approximations are widely used for summarizing posterior distributions in Bayesian models, especially in high-dimensional settings. However, a drawback of such approximations is the inability to capture skewness or more complex features of the posterior. Recent work suggests applying skewness corrections to existing Gaussian or other symmetric approximations to address this limitation. We propose to incorporate the skewness correction into the definition of an approximating variational family. We consider approximating the posterior for hierarchical models, in which there are ``global'' and ``local'' parameters. A baseline variational approximation is defined as the product of a Gaussian marginal posterior for global parameters and a Gaussian conditional posterior for local parameters given the global ones. Skewness corrections are then considered. The adjustment of the conditional posterior term for local variables is adaptive to the global parameter value. Optimization of baseline variational parameters is performed jointly with the skewness correction. Our approach allows the location, scale and skewness to be captured separately, without using additional parameters for skewness adjustments. The proposed method substantially improves accuracy for only a modest increase in computational cost compared to state-of-the-art Gaussian approximations. Good performance is demonstrated in generalized linear mixed models and multinomial logit discrete choice models.

Problem

• Correcting skewness in hierarchical Bayesian model posteriors
• Adaptive conditional scale and skewness adjustments for local parameters
• Improving variational inference accuracy without significant computational cost

Innovation

• Skewness correction in variational family definition
• Adaptive conditional posterior adjustment for local variables
• Joint optimization of baseline and skewness parameters