🤖 AI Summary
This work addresses the lack of a unified theoretical framework and failure mode analysis for existing batch Bayesian optimization methods—such as Constant Liar (CL), Kriging Believer (KB), and fantasy models. We introduce the concept of “efficient conditioning,” showing that these approaches are specific instantiations of a common mechanism, and reveal their intrinsic connections to local penalization and determinantal point processes. Building on this insight, we develop a Structural Diversity Diagnostic (SDD) method and prove that Gaussian processes under this framework generate batch points with theoretically guaranteed diversity. Experiments demonstrate that the implicit penalization in CL/KB matches or outperforms explicit penalization schemes; efficient conditioning achieves performance comparable to joint qEI in high-dimensional tasks and extends effectively to Multiquadric RBF networks; meanwhile, parametric models like random forests often induce batch collapse, whereas neural networks restore diversity at substantial computational cost.
📝 Abstract
Constant Liar (CL), Kriging Believer (KB), and fantasy models are widely used for batch selection in parallel Bayesian Optimization, yet a unified theory explaining their effectiveness and conditions under which they fail has been lacking. We identify efficient conditioning as the key surrogate property the ability to update predictions in closed form when data is augmented. We prove that Gaussian Processes satisfy this requirement, producing provably distinct batch points with separation of order l, and that this holds for any acquisition function monotonically non decreasing in posterior uncertainty (EI, UCB, PI), with qualitatively similar behavior for Thompson Sampling. We unify CL, KB, and fantasy models as instances of a single conditioning mechanism differing only in the lie value distribution, and draw quantitative connections to Local Penalization (LP) and qualitative connections to Determinantal Point Processes (DPPs). To disentangle model structure from optimizer randomness, we introduce the Structural Diversity Diagnostic (SDD), a reusable methodology for testing surrogate compatibility. Experiments on Hartmann6D, Ackley 8D, Levy10D, and SVM hyperparameter tuning validate all theoretical predictions: CL or KBs implicit penalty matches or outperforms explicit LP greedy conditioning achieves convergence on par with joint qEI efficient conditioning extends to Multiquadric RBF networks; and parametric surrogates produce degenerate batches even when fully retrained (random forests), while neural networks regain diversity only at 15x the wall clock cost of GP conditioning. Robustness is confirmed across multiple initial datasets and under observation noise.