ECom-Bench: Can LLM Agent Resolve Real-World E-commerce Customer Support Issues?

2025-07-07

AI Summary

Existing large language model (LLM)-based agents lack systematic, multimodal evaluation frameworks tailored to real-world e-commerce customer service scenarios. Method: We introduce EC-Bench—the first multimodal agent benchmark specifically designed for e-commerce customer service—constructed from millions of real-world dialogues. It features a user-profile-driven dynamic simulation mechanism and a high-difficulty composite task suite encompassing cross-modal understanding, multi-turn reasoning, and real-time decision-making. Contribution/Results: EC-Bench significantly enhances evaluation authenticity and challenge. Experiments reveal that state-of-the-art multimodal models (e.g., GPT-4o) achieve only 10–20% pass@3 accuracy, exposing critical gaps in operational capability. This work shifts e-commerce AI agent evaluation from isolated skill assessment toward end-to-end problem-solving performance. The benchmark code and data will be publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Agent Communication
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

In this paper, we introduce ECom-Bench, the first benchmark framework for evaluating LLM agent with multimodal capabilities in the e-commerce customer support domain. ECom-Bench features dynamic user simulation based on persona information collected from real e-commerce customer interactions and a realistic task dataset derived from authentic e-commerce dialogues. These tasks, covering a wide range of business scenarios, are designed to reflect real-world complexities, making ECom-Bench highly challenging. For instance, even advanced models like GPT-4o achieve only a 10-20% pass^3 metric in our benchmark, highlighting the substantial difficulties posed by complex e-commerce scenarios. Upon publication, the code and data will be open-sourced to facilitate further research and development in this domain.

Problem

• Evaluating LLM agents in e-commerce customer support
• Assessing multimodal capabilities in real-world scenarios
• Addressing complex e-commerce business challenges

Innovation

• Multimodal LLM agent for e-commerce support
• Dynamic user simulation with real personas
• Open-source benchmark with complex tasks