EcoAgent-Bench: Evaluating Economic Decision-Making in Budget-Constrained LLM Agents

2026-08-05

AI Summary

This work addresses a critical gap in existing agent evaluation frameworks, which typically ignore resource costs and thus fail to capture the economic trade-offs inherent in real-world deployment. To bridge this gap, we introduce EcoAgent-Bench—the first benchmark that internalizes budget constraints and action costs as core evaluation dimensions—featuring 304 tasks across five categories derived from GAIA, HotpotQA, and MuSiQue. We evaluate seven LLM-based agents and four control policies in both tool-API and workspace-CLI environments. Introducing an economic consistency score, we expose the limitations of optimizing solely for micro-average accuracy and analyze budget sensitivity via threshold scanning. Our experiments reveal that state-of-the-art agents achieve micro strict success rates of only 3.9–24.0%, with economic consistency peaking at just 7.3%; notably, GPT-5.4 shows negligible improvement in upgrade rate despite increased budgets, highlighting a severe disconnect between task completion capability and economic decision-making.

Abstract

Agent benchmarks usually measure task completion and treat resource use as an auxiliary statistic. In deployment, however, the choice among a local lookup, broad search, composite research tool, stronger model, or human escalation is part of the task itself. We introduce EcoAgent-Bench, in which every task specifies priced actions and an explicit budget. Its 304 real-derived tasks span five families adapted from GAIA, HotpotQA, and MuSiQue, and test four decisions: avoiding unnecessary escalation, escalating when local evidence is insufficient, selecting a model tier, and stopping on unsupported premises. We evaluate seven LLM agents in tool-API and workspace-CLI settings, together with four oracle scripted controls. Micro-averaged accuracy rewards one-sided policies: always-escalate controls achieve high micro success while failing save-oriented tasks. We therefore also report an economic-consistency score (the worse of accuracy on upgrade-oriented and save-oriented family groups) which exposes this failure. Tool-API agents attain only 3.9-24.0% micro strict success (at most 7.3% economic consistency), often either stopping before warranted escalation or overspending on cheap tasks. A threshold-crossing budget sweep changes GPT-5.4's escalation rate from 0% to only 3%. These results show that completion under a budget and economical action selection are distinct properties. We release the task bundle, transformation pipeline, frozen evaluation environments, and integrity-bound result artifacts needed to study both.

Problem

• budget-constrained agents
• economic decision-making
• LLM evaluation
• resource allocation
• agent benchmarking

Innovation

• budget-constrained reasoning
• economic decision-making
• LLM agent benchmarking
• economic-consistency score
• priced actions