Bayesian Optimization with Fisher Information Geometry: Gradient Bounds and Trust-Region Methods

2026-09-25

AI Summary

This study addresses the vanishing gradient problem in acquisition functions and the lack of theoretical justification for heuristic strategies in high-dimensional Bayesian optimization. By analyzing gradient upper bounds through Fisher information geometry, this work proposes the FITR algorithm. Specifically, FITR replaces conventional length-scale scaling with local pullback Fisher weights, thereby overcoming the limitation of requiring explicit length scales in Gaussian process kernels. This approach naturally accommodates non-isotropic surrogate models and provides a unified theoretical framework for heuristic methods such as RAASP. Experimental results demonstrate that FITR significantly mitigates gradient vanishing in high-dimensional settings across standard benchmarks, exhibiting superior performance and strong generalization capabilities.

Abstract

We study Bayesian optimization (BO) through the lens of information geometry. Pulling back the Fisher information metric through the surrogate posterior map yields a local sensitivity tensor on the input space, which leads to an upper bound on the gradient of reparameterizable acquisition functions. This view explains vanishing-gradient behavior in high-dimensional BO and provides a common interpretation of heuristics such as RAASP and dimension-scaled lengthscales. Building on this analysis, we propose FITR, a trust-region-based BO method that replaces lengthscale-based scaling by local pullback-Fisher weights. FITR is not restricted to GP kernels with explicit lengthscales. On GP benchmarks with an SE kernel, experiments show competitive performance using FITR. The proposed method also easily generalizes to non-isotropic surrogates, although the gains are more task-dependent in that setting.

Problem

• Bayesian Optimization
• High-dimensional
• Vanishing Gradient
• Information Geometry
• Trust-Region

Innovation

• Bayesian Optimization
• Fisher Information Geometry
• Trust-Region Methods
• Gradient Bounds
• Pullback Metric