EcomScriptBench: A Multi-task Benchmark for E-commerce Script Planning via Step-wise Intention-Driven Product Association

2025-05-21

AI Summary

Large language models (LLMs) struggle to jointly perform script planning and precise product retrieval in e-commerce scenarios, particularly due to intent-action semantic misalignment. Method: We propose EcomScript, the first end-to-end script planning task for goal-driven, multi-step shopping action generation with step-wise product association. To address intent-action mismatch, we introduce an intent-driven, stepwise product matching paradigm that aligns user intent and actions via semantic similarity and jointly models script generation and product retrieval. Contribution/Results: We construct EcomScriptBench—the first large-scale benchmark derived from real e-commerce platform data—containing 605K scripts, 2.4M products, and a human-annotated subset. Experiments show that mainstream (L)LMs underperform significantly on this task; explicitly modeling purchase intent yields substantial improvements. This work establishes a novel task definition, a principled methodological framework, and a unified evaluation benchmark for e-commerce agent research.

Technology Category

• Multiagent Systems: Multiagent Planning
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Goal-oriented script planning, or the ability to devise coherent sequences of actions toward specific goals, is commonly employed by humans to plan for typical activities. In e-commerce, customers increasingly seek LLM-based assistants to generate scripts and recommend products at each step, thereby facilitating convenient and efficient shopping experiences. However, this capability remains underexplored due to several challenges, including the inability of LLMs to simultaneously conduct script planning and product retrieval, difficulties in matching products caused by semantic discrepancies between planned actions and search queries, and a lack of methods and benchmark data for evaluation. In this paper, we step forward by formally defining the task of E-commerce Script Planning (EcomScript) as three sequential subtasks. We propose a novel framework that enables the scalable generation of product-enriched scripts by associating products with each step based on the semantic similarity between the actions and their purchase intentions. By applying our framework to real-world e-commerce data, we construct the very first large-scale EcomScript dataset, EcomScriptBench, which includes 605,229 scripts sourced from 2.4 million products. Human annotations are then conducted to provide gold labels for a sampled subset, forming an evaluation benchmark. Extensive experiments reveal that current (L)LMs face significant challenges with EcomScript tasks, even after fine-tuning, while injecting product purchase intentions improves their performance.

Problem

• LLMs struggle with e-commerce script planning and product retrieval simultaneously
• Semantic gaps hinder product matching in planned shopping actions
• Lack of benchmarks for evaluating e-commerce script generation methods

Innovation

• Multi-task benchmark for e-commerce script planning
• Step-wise intention-driven product association framework
• Large-scale dataset with human-annotated gold labels