Proximal Interacting Particle Langevin Algorithms

2024-06-20

AI Summary

Bayesian inference for nonsmooth latent-variable models with non-differentiable joint densities remains challenging due to the failure of standard gradient-based MCMC methods. Method: This paper proposes the Proximal Interacting Particle Langevin Algorithm (PIPLA), the first framework unifying proximal MCMC with interacting particle Langevin dynamics to jointly perform posterior sampling and parameter estimation. Contributions/Results: Theoretically, we derive non-asymptotic error bounds and establish convergence of parameter estimates under strong log-concavity. Methodologically, PIPLA is systematically extended to sparse Bayesian logistic regression, neural network training with nonsmooth activation functions (e.g., ReLU), image deblurring, and sparse matrix completion. Empirical results demonstrate PIPLA’s superior accuracy, stability, and sparsity recovery over existing approaches. By bridging proximal optimization and particle-based sampling, PIPLA provides a theoretically grounded and practically effective framework for nonsmooth Bayesian inference.

Technology Category

• Machine Learning: Probabilistic Circuits and Graphical Models
• Reasoning under Uncertainty: Relational Probabilistic Models
• Search and Optimization: Non-convex Optimization

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We introduce a class of algorithms, termed proximal interacting particle Langevin algorithms (PIPLA), for inference and learning in latent variable models whose joint probability density is non-differentiable. Leveraging proximal Markov chain Monte Carlo techniques and interacting particle Langevin algorithms, we propose three algorithms tailored to the problem of estimating parameters in a non-differentiable statistical model. We prove nonasymptotic bounds for the parameter estimates produced by the different algorithms in the strongly log-concave setting and provide comprehensive numerical experiments on various models to demonstrate the effectiveness of the proposed methods. In particular, we demonstrate the utility of our family of algorithms for sparse Bayesian logistic regression, training of sparse Bayesian neural networks or neural networks with non-differentiable activation functions, image deblurring, and sparse matrix completion. Our theory and experiments together show that PIPLA family can be the de facto choice for parameter estimation problems in non-differentiable latent variable models.

Problem

• Estimating parameters in non-differentiable statistical models
• Handling latent variable models with non-differentiable densities
• Addressing inference in sparse or non-differentiable neural networks

Innovation

• Proximal MCMC techniques for non-differentiable models
• Interacting particle Langevin algorithms integration
• Parameter estimation in non-differentiable latent models