Dimension reduction via score ratio matching

2024-10-25

arXiv.org

AI Summary

This work addresses the challenge of Bayesian dimensionality reduction in gradient-free settings—such as purely data-driven or simulation-based inference—where gradients of the target posterior are unavailable. To this end, we propose a score-ratio-based Bayesian dimensionality reduction framework that operates without explicit gradient computation. Our method innovatively extends score matching to gradient-deficient scenarios for dimensionality reduction; introduces a dedicated neural network architecture that jointly learns the score-ratio function and identifies an informative low-dimensional subspace; and incorporates manifold-aware regularization alongside an iterative basis-optimization algorithm leveraging eigenvalue truncation. Evaluated on PDE-constrained Bayesian inverse problems and conditional generation tasks, the approach achieves high-fidelity low-dimensional posterior approximations using only limited simulation data, significantly outperforming standard score matching and other gradient-free dimensionality reduction baselines.

Technology Category

Machine Learning: Learning with ManifoldsSearch and Optimization: Sampling/Simulation-based SearchComputer Vision: Learning & Optimization for CV

Application Category

Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingUser Modeling, Personalization and Recommendation: Fairness-aware retrieval and rankingGraph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Gradient-based dimension reduction decreases the cost of Bayesian inference and probabilistic modeling by identifying maximally informative (and informed) low-dimensional projections of the data and parameters, allowing high-dimensional problems to be reformulated as cheaper low-dimensional problems. A broad family of such techniques identify these projections and provide error bounds on the resulting posterior approximations, via eigendecompositions of certain diagnostic matrices. Yet these matrices require gradients or even Hessians of the log-likelihood, excluding the purely data-driven setting and many problems of simulation-based inference. We propose a framework, derived from score-matching, to extend gradient-based dimension reduction to problems where gradients are unavailable. Specifically, we formulate an objective function to directly learn the score ratio function needed to compute the diagnostic matrices, propose a tailored parameterization for the score ratio network, and introduce regularization methods that capitalize on the hypothesized low-dimensional structure. We also introduce a novel algorithm to iteratively identify the low-dimensional reduced basis vectors more accurately with limited data based on eigenvalue deflation methods. We show that our approach outperforms standard score-matching for problems with low-dimensional structure, and demonstrate its effectiveness for PDE-constrained Bayesian inverse problems and conditional generative modeling.

Problem

• Extends gradient-based dimension reduction to gradient-unavailable problems
• Learns score ratio function for diagnostic matrices without gradients
• Improves accuracy of low-dimensional basis identification with limited data

Innovation

• Score ratio matching for gradient-free dimension reduction
• Tailored parameterization for score ratio networks
• Iterative eigenvalue deflation for basis identification