Adaptive Linear Embedding for Nonstationary High-Dimensional Optimization

2025-05-16

AI Summary

High-dimensional Bayesian optimization suffers from the curse of dimensionality and the failure of the global low-dimensional assumption—especially when the objective exhibits nonstationarity, spatially varying effective dimensionality, and heteroscedasticity—causing significant performance degradation in methods like REMBO. To address this, we propose SA-REMBO, an adaptive multi-embedding Bayesian optimization framework. SA-REMBO introduces an index-conditioned product-kernel Gaussian process that dynamically selects embedding structures based on input location; jointly models index variables and latent subspaces to enable adaptive local Gaussian linear embeddings; and provides the first unified characterization of local effective dimensionality, nonstationarity, and heteroscedasticity. On synthetic and real-world high-dimensional benchmarks, SA-REMBO achieves up to 3.2× faster convergence than state-of-the-art low-rank BO methods (e.g., REMBO), markedly improving robustness for strongly nonstationary black-box optimization.

Technology Category

Search and Optimization: Mixed Discrete/Continuous SearchReasoning under Uncertainty: Stochastic OptimizationIntelligent Robots: Learning & Optimization for ROB

Application Category

Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphsSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingUser Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Bayesian Optimization (BO) in high-dimensional spaces remains fundamentally limited by the curse of dimensionality and the rigidity of global low-dimensional assumptions. While Random EMbedding Bayesian Optimization (REMBO) mitigates this via linear projections into low-dimensional subspaces, it typically assumes a single global embedding and a stationary objective. In this work, we introduce Self-Adaptive embedding REMBO (SA-REMBO), a novel framework that generalizes REMBO to support multiple random Gaussian embeddings, each capturing a different local subspace structure of the high-dimensional objective. An index variable governs the embedding choice and is jointly modeled with the latent optimization variable via a product kernel in a Gaussian Process surrogate. This enables the optimizer to adaptively select embeddings conditioned on location, effectively capturing locally varying effective dimensionality, nonstationarity, and heteroscedasticity in the objective landscape. We theoretically analyze the expressiveness and stability of the index-conditioned product kernel and empirically demonstrate the advantage of our method across synthetic and real-world high-dimensional benchmarks, where traditional REMBO and other low-rank BO methods fail. Our results establish SA-REMBO as a powerful and flexible extension for scalable BO in complex, structured design spaces.

Problem

• Addresses high-dimensional optimization limitations in Bayesian Optimization
• Generalizes REMBO to support multiple adaptive local embeddings
• Captures nonstationary and heteroscedastic objective landscapes effectively

Innovation

• Multiple random Gaussian embeddings for local subspaces
• Index-conditioned product kernel in Gaussian Process
• Adaptive embedding selection for nonstationary objectives