Benchmarking Prompt Optimization of Large Language Models With Chess

2026-09-30

AI Summary

This study addresses the saturation of large language model benchmarks, data contamination, and the prohibitive cost and poor reproducibility of evaluating automatic prompt optimization (APO). To overcome these challenges, this work proposes a chess-puzzle-based APO benchmark. Methodologically, an anti-contamination dataset is constructed using Lichess data integrated with engine evaluations and game replays. By freezing model weights and optimizing prompts exclusively, the approach enables deterministic scoring and dynamic difficulty adjustment. Experimental comparisons across six APO algorithms demonstrate that the proposed benchmark exhibits high discriminative power, with the strongest model achieving only a 55% solve rate, thereby effectively revealing performance disparities among different methods. Furthermore, the overall evaluation cost is reduced to approximately 800 USD, establishing a reproducible and cost-effective assessment paradigm for APO research.

Abstract

Evaluating large language models becomes increasingly challenging as their capabilities advance: benchmarks can saturate, public test sets risk contamination, and assessing harder tasks can require expensive grading or execution infrastructure. These challenges are amplified in automatic prompt optimization (APO), where evaluation is repeated throughout the search for better prompts. Studying APO therefore requires a benchmark that is cheap and deterministic to score, hard enough to leave room for improvement, and renewable as models evolve. We introduce a chess benchmark built from 1,118 Lichess puzzles to study APO for frozen LLMs: we optimize their prompts without updating their model weights. Chess combines inexpensive exact-match scoring, engine-based evaluation of alternative moves, and a renewable supply of problems with adjustable difficulty. Unlike evaluations that report only success on isolated test items, the benchmark also connects puzzle-solving gains to short game-play rollouts within the same domain. We use it to evaluate six APO algorithms on eight target models, measuring not only baseline strength but also how much each model responds to optimization and whether optimized prompts transfer across models and to game play. Chess is thus a well-suited benchmark for APO: it is (i) challenging, as even the strongest evaluated model, Gemini 3.5 Flash (used as the meta-model), solves only about 55\% of puzzles; (ii) discriminative, revealing gains, unchanged performance, and regressions across methods and models; (iii) renewable, with fresh puzzles to reduce contamination risk and adjustable difficulty to maintain headroom as models improve; and (iv) affordable, as the complete study runs for around \$800. We release the puzzles, optimization and evaluation code, and dataset-renewal scripts (https://github.com/imec-ailabs/Automatic-Prompt-Optimization-with-Chess).

Problem

• Large Language Models
• Automatic Prompt Optimization
• Benchmarking
• Evaluation
• Chess

Innovation

• Automatic Prompt Optimization
• Large Language Models
• Chess Benchmark
• Prompt Transferability
• Renewable Evaluation