MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations

2026-07-30

AI Summary

This work addresses a critical gap in evaluating large language model (LLM) agents: the lack of effective assessment of long-term consistency—the ability to maintain goal-directed behavior and adapt decisions based on accumulated evidence over extended time horizons. To this end, we propose the first long-term consistency evaluation framework tailored for e-commerce operations, featuring a 365-day order-level simulation environment built from 98,843 real product records. The framework encompasses interdependent cyclical decisions including procurement, listing, pricing, and cash flow management, and incorporates an asynchronous feedback mechanism linking upstream actions to downstream outcomes, requiring agents to retrospectively revise past decisions. Evaluated across 48 full-year simulations with eight prominent LLMs, 26 integrated tools, and two agent architectures, the best-performing configuration achieves only 27.3% of the average final net worth attained by human participants, revealing a substantial gap between current LLMs and human-level long-term coherent decision-making.

Abstract

Large language model agents are increasingly evaluated as autonomous tool users, yet most benchmarks focus on bounded tasks with immediate success criteria. Real-world deployments often require Long-Term Coherence, the capacity to preserve purposeful behavior across extended horizons while adapting decisions to accumulated evidence. Evaluating this capacity requires a persistent environment in which actions constrain future choices, feedback arrives at heterogeneous delays, and incoherent behavior produces measurable cumulative effects. Seller-side e-commerce provides a suitable setting for this evaluation through recurrent and interdependent decisions over Product Sourcing, Listing and Pricing Control, Cash-Flow Management, and Mixed-Latency Feedback Adaptation. We introduce MerchantBench, a 365-day order-level simulation grounded in 98,843 real e-commerce product records and equipped with 26 tools for agent interaction. MerchantBench couples promptly observable Upstream Supplier Events with delayed Downstream Order Outcomes, requiring agents to follow individual order lifecycles and revisit earlier decisions. We evaluate eight LLMs under two agent frameworks in 48 runs, each spanning 365 simulated days. Our results reveal a substantial gap between even the latest LLMs and human participants, with the best LLM configuration attaining only 27.3\% of the mean final net assets achieved by human participants.

Problem

• Long-Term Coherence
• LLM Agents
• E-Commerce Operations
• Benchmarking
• Delayed Feedback

Innovation

• Long-Term Coherence
• LLM Agents
• E-Commerce Simulation
• Mixed-Latency Feedback
• Persistent Environment