E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios

2026-07-26

AI Summary

Existing benchmarks struggle to evaluate large language models’ ability to perform multi-step tool use, uncover hidden information, and alter system states in realistic product scenarios. To address this gap, this work proposes E-Bench—a fully synthetic benchmark comprising 323 state-changing tasks across three product domains. E-Bench decouples environment and task generation through a graph-guided database population strategy and an asymmetric generator-solver mechanism, yielding product environments without isolated data and tasks that feature both informational and tool-based gaps. Evaluation is made deterministic via database state differencing. Experimental results show that even the strongest models achieve below 60% on the Pass³ metric, and incorporating code execution (E-Bench-Code) only marginally improves reliability to under 70%, underscoring the persistent challenge of robust multi-step tool use.

Abstract

Large Language Models (LLMs) are increasingly deployed as agents that interact with stateful environments over multiple steps: gathering hidden information, composing tool calls, and committing state changes. We refer to this capability as multi-step tool use. Existing benchmarks have advanced tool-use agent evaluation, but often focus on isolated API calls, short trajectories, or settings that are difficult to scale or control. We introduce E-Bench, a fully synthetic benchmark with 323 state-changing tasks across three product domains: Honor of Kings, QQ Music, and Tencent Meeting. E-Bench decouples environment synthesis from task synthesis: graph-guided database filling builds reusable, orphan-free product environments, while generator-solver asymmetry creates tasks with both an information gap and a tool gap, requiring agents to discover hidden data and compose multiple tool calls before changing state. Outcomes are graded deterministically by database-state diffs. Since both environments and tasks are synthetic, E-Bench is controllable at the environment level and scalable at the task level. Benchmarking 11 cutting-edge LLMs shows that multi-step tool use remains challenging: Pass^3 stays below 60% for the strongest models, and even with code execution in the E-Bench-Code extension, reliability (Pass^3) remains below 70%.

Problem

• multi-step tool use
• LLM agents
• benchmarking
• stateful environments
• real-world product scenarios

Innovation

• multi-step tool use
• synthetic benchmark
• graph-guided environment synthesis
• generator-solver asymmetry
• stateful agent evaluation