Clustering-based Meta Bayesian Optimization with Theoretical Guarantee

2025-03-08

AI Summary

In multi-task black-box optimization, the strong heterogeneity and large scale of historical tasks degrade both performance and scalability of meta-Bayesian optimization (Meta-BO). Method: This paper proposes a clustering-based Meta-BO framework that jointly integrates manifold-geometric prototype modeling with a kernelized statistical distance adaptive weighting mechanism, coupled with spectral clustering and online Bayesian updating—ensuring theoretical convergence while enhancing robustness to task heterogeneity. Contribution/Results: Theoretical analysis establishes an adaptive regret upper bound that contracts with increasing task similarity. Empirically, on real-world hyperparameter optimization benchmarks, the method achieves a 37% faster convergence than state-of-the-art Meta-BO approaches, significantly improving scalability and generalization across diverse tasks.

Technology Category

• Search and Optimization: Metareasoning and Metaheuristics
• Machine Learning: Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work

Abstract

Bayesian Optimization (BO) is a well-established method for addressing black-box optimization problems. In many real-world scenarios, optimization often involves multiple functions, emphasizing the importance of leveraging data and learned functions from prior tasks to enhance efficiency in the current task. To expedite convergence to the global optimum, recent studies have introduced meta-learning strategies, collectively referred to as meta-BO, to incorporate knowledge from historical tasks. However, in practical settings, the underlying functions are often heterogeneous, which can adversely affect optimization performance for the current task. Additionally, when the number of historical tasks is large, meta-BO methods face significant scalability challenges. In this work, we propose a scalable and robust meta-BO method designed to address key challenges in heterogeneous and large-scale meta-tasks. Our approach (1) effectively partitions transferred meta-functions into highly homogeneous clusters, (2) learns the geometry-based surrogate prototype that capture the structural patterns within each cluster, and (3) adaptively synthesizes meta-priors during the online phase using statistical distance-based weighting policies. Experimental results on real-world hyperparameter optimization (HPO) tasks, combined with theoretical guarantees, demonstrate the robustness and effectiveness of our method in overcoming these challenges.

Problem

• Addresses scalability in meta-BO with large historical tasks.
• Handles heterogeneous functions in meta-learning for optimization.
• Improves convergence to global optimum via clustering and meta-priors.

Innovation

• Partitions meta-functions into homogeneous clusters
• Learns geometry-based surrogate prototypes per cluster
• Adaptively synthesizes meta-priors using statistical weighting