🤖 AI Summary
Existing benchmarks struggle to evaluate the ability of large language models to perform end-to-end optimization tasks in real-world business settings. This work proposes the first comprehensive end-to-end evaluation benchmark that spans business requirement interpretation, mathematical modeling, algorithm selection, code implementation, and report generation. It introduces three key innovations: business-semantic anti-template traps, cross-module consistency checks, and a dual-layer ORAC validity verification framework, covering core optimization paradigms such as integer programming, robust optimization, stochastic programming, and non-convex optimization. Experiments reveal systematic deficiencies in current models—including omitted constraints and inconsistencies between formulated models and generated code—that remain undetected under conventional single-metric evaluations, thereby demonstrating the necessity and effectiveness of this benchmark for assessing complex, multi-stage optimization workflows.
📝 Abstract
LLM-based agents are increasingly deployed to solve optimization problems, yet existing benchmarks evaluate them on pre-structured mathematical formulations that bypass the most critical challenge: translating complex business requirements into correct models and solve efficiently. We introduce Opti-Agent-Bench, an end-to-end benchmark that evaluates Large Language Models (LLMs) across the complete optimization R&D pipeline, from understanding business-language descriptions through mathematical modeling, algorithm selection, and code implementation, to solution report generation. Our design rests on three pillars: (1) businesssemantic authenticity with anti-template traps that defeat pattern matching; (2) modular evaluation with cross-module consistency checking across Problem Understanding, Formal Modeling, Implementation, and Reporting; and (3) the ORAC bi-level validity framework that simultaneously ensures task quality and scoring integrity. Across several industrialscale tasks spanning integer programming, robust optimization, stochastic programming, and non-convex optimization, we expose critical failure modes of current models, including constraint omission, model-code inconsistency, and report-implementation divergence, that remain invisible under conventional single-metric evaluation.