Automated Query-Product Relevance Labeling using Large Language Models for E-commerce Search

2025-02-21

AI Summary

To address the high cost, low efficiency, and error-proneness of manual query-item relevance annotation in e-commerce search, this paper proposes an automated annotation framework leveraging large language models (LLMs), specifically LLaMA and GPT. We present the first systematic empirical validation that LLMs can achieve human-expert-level performance on large-scale relevance judgment tasks. Our method introduces a novel multi-strategy prompting framework integrating chain-of-thought (CoT) reasoning, in-context learning (ICL), and retrieval-augmented generation with maximal marginal relevance (RAG-MMR). Evaluated across multiple public and proprietary datasets, our approach achieves annotation accuracy within ±1.2% of human benchmarks while improving throughput by over 100×. The framework has been deployed in production for training and iterative evaluation of search ranking models, significantly reducing annotation costs and accelerating R&D cycles.

Technology Category

Search and Optimization: Learning to SearchMachine Learning: Large Multimodal Models (LMMs)Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labelingSearch and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved informationUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Accurate query-product relevance labeling is indispensable to generate ground truth dataset for search ranking in e-commerce. Traditional approaches for annotating query-product pairs rely on human-based labeling services, which is expensive, time-consuming and prone to errors. In this work, we explore the application of Large Language Models (LLMs) to automate query-product relevance labeling for large-scale e-commerce search. We use several publicly available and proprietary LLMs for this task, and conducted experiments on two open-source datasets and an in-house e-commerce search dataset. Using prompt engineering techniques such as Chain-of-Thought (CoT) prompting, In-context Learning (ICL), and Retrieval Augmented Generation (RAG) with Maximum Marginal Relevance (MMR), we show that LLM's performance has the potential to approach human-level accuracy on this task in a fraction of the time and cost required by human-labelers, thereby suggesting that our approach is more efficient than the conventional methods. We have generated query-product relevance labels using LLMs at scale, and are using them for evaluating improvements to our search algorithms. Our work demonstrates the potential of LLMs to improve query-product relevance thus enhancing e-commerce search user experience. More importantly, this scalable alternative to human-annotation has significant implications for information retrieval domains including search and recommendation systems, where relevance scoring is crucial for optimizing the ranking of products and content to improve customer engagement and other conversion metrics.

Problem

• Automate query-product relevance labeling
• Enhance e-commerce search efficiency
• Reduce human-labeling cost and time

Innovation

• LLMs automate query-product relevance labeling
• Prompt engineering enhances LLM performance
• Scalable labeling improves e-commerce search efficiency