Harnessing Large Language Models to Compile Task-Relevant Context into Bayesian Optimisation

2026-09-29

AI Summary

This study addresses the challenge of effectively integrating contextual information, such as domain knowledge, into Bayesian optimization. To this end, we propose HarBO, a framework that introduces the first "LLM-compiled" paradigm for Bayesian optimization. By leveraging the code generation capabilities of large language models, our approach compiles general context into executable core components of Bayesian optimization. Furthermore, we construct a multi-stage verification workflow and provide theoretical regret bound analysis under imperfect compilation conditions. Experimental results demonstrate that HarBO achieves performance competitive with task-specific methods on both synthetic and real-world benchmark tasks, validating the effectiveness of this paradigm in incorporating domain priors.

Abstract

Incorporating rich task-relevant context, such as domain knowledge and external observations, is a key capability yet remains challenging for Bayesian optimisation (BO). Recently, practitioners have started to use large language models (LLMs) to generate and execute BO programs through coding harnesses. In such emerging practices, the posterior belief is shaped not only by Bayesian inference but also by LLM-generated model and data artefacts, offering a flexible route for task context to enter BO as executable code. To study whether and how LLMs can be harnessed to compile diverse contextual signals for BO, we formulate LLM-compiled BO as generalised-context decision making. We propose HarBO, a BO-specialised harness that compiles generalised context into the core artefacts of standard BO through a validated multi-stage workflow. Our theory analyses the regret under imperfect compilation and the effect of adding new context. Across synthetic functions and real-world benchmarks, we find that LLM harnesses can effectively compile context into standard BO, achieving competitive performance with specialised LLM-embedding-based and direct LLM-in-the-loop BO methods. General coding harnesses can be effective in familiar domains such as hyperparameter optimisation, but fall short in unfamiliar, context-rich domains. Together, these results establish LLM harnesses as a promising, but not automatically reliable, route for making rich task context usable in BO.

Problem

• Bayesian Optimisation
• Large Language Models
• Task-Relevant Context
• Context Compilation

Innovation

• Bayesian Optimisation
• Large Language Models
• Context Compilation
• Regret Analysis
• HarBO