Flow-Transformed Implicit Processes for Function-Space Variational Inference

2026-06-01

AI Summary

Implicit process priors, lacking closed-form expressions, pose significant challenges for Bayesian posterior inference in function space, and conventional Gaussian variational approximations often fail to capture asymmetric, heavy-tailed, or multimodal uncertainties. This work proposes a novel function-space variational inference approach that constructs a tractable prior approximation via a finite set of sampled functions and, for the first time, employs normalizing flows to flexibly model the combination weights, thereby replacing the restrictive Gaussian assumption. This method substantially enhances the expressiveness of the approximate posterior while preserving computational feasibility, effectively recovering complex posterior structures overlooked by Gaussian approximations and yielding more accurate uncertainty quantification across multiple tasks.

Abstract

Implicit-process priors define distributions over functions through flexible generative mechanisms, making them attractive for Bayesian function-space modelling. However, performing posterior inference with such priors is challenging because their induced function-space distributions are typically not available in closed form. One practical strategy is to approximate the prior using a finite collection of sampled functions, and then represent posterior functions as learned combinations of these samples. Existing approaches commonly place a Gaussian variational distribution over the combination weights. While tractable, this choice limits the shapes of posterior uncertainty that can be represented, especially when the true posterior is asymmetric, heavy-tailed, or multimodal. We propose Flow-Transformed Implicit Processes (FTIP), a variational inference method that makes this finite-dimensional function-space approximation more expressive. Instead of using a Gaussian distribution over the combination weights, FTIP uses a normalizing flow to define a richer variational distribution. This induces a flexible posterior distribution over functions while preserving tractable optimization. We train the model using a Black-Box α objective, allowing us to compare mass-covering and mode-seeking variational behaviour. Experiments show that FTIP captures asymmetric and multimodal posterior structure in function space that Gaussian coefficient approximations tend to smooth or collapse.

Problem

• implicit processes
• function-space inference
• variational inference
• posterior approximation
• non-Gaussian uncertainty

Innovation

• normalizing flow
• function-space variational inference
• implicit processes
• multimodal posterior
• Bayesian nonparametrics