AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers

2026-07-31

AI Summary

Existing benchmarks struggle to evaluate the ability of large language model (LLM) agents to perform sequential decision-making in hyperparameter optimization based on experimental feedback. To address this gap, this work proposes the first evaluation paradigm specifically designed for LLM agents in sequential hyperparameter optimization, introducing a serialized benchmark comprising 30 executable machine learning tasks that require agents to iteratively generate effective new configurations using historical hyperparameters, metrics, and logs. We evaluate 12 prominent LLM agents alongside traditional baselines under a unified protocol across seven task categories. Results indicate that current LLM agents exhibit nascent cross-domain optimization capabilities but still face significant limitations in iterative refinement, complex log diagnostics, and consistently converging toward optimal performance, thereby bridging the evaluation gap between static code generation and dynamic experimental optimization.

Abstract

As LLMs evolve from code completion systems into autonomous scientific agents, evaluating their ability to conduct experiments has become increasingly important. Existing benchmarks typically focus on static code generation, paper replication, or final answer correctness, but do not directly assess whether agents can interpret experimental evidence and use it to guide subsequent hyperparameter decisions. To address this gap, we introduce AgentHPOBench, a sequential benchmark comprising 30 executable machine learning tasks across seven research categories. Each task begins with a validated baseline run, after which an agent performs several sequential interventions. At each step, the agent observes the accumulated configurations, metrics, and logs before proposing the next valid configuration. We evaluate 12 widely used agents and conventional HPO baselines under a unified protocol. The results show that current agents exhibit measurable experimental optimization ability across domains, but still face clear limitations in sustained iterative refinement, complex log diagnosis, and consistent progress toward reported reference performance.

Problem

• LLM agents
• hyperparameter optimization
• sequential decision-making
• experimental evaluation
• benchmarking

Innovation

• LLM agents
• sequential hyperparameter optimization
• AgentHPOBench
• experimental reasoning
• iterative refinement