🤖 AI Summary
Manual relevance annotation in personalized search is costly and poorly scalable. Method: We propose an automated relevance assessment framework based on fine-tuned large language models (LLMs), incorporating query-document semantic matching and context-aware discrimination, trained via supervised fine-tuning on high-quality human annotations. Contribution/Results: This is the first work to achieve high inter-annotator agreement (Cohen’s κ > 0.85) between LLMs and human annotators in a large-scale production search system (Pinterest). The method triples query coverage, reduces the minimum detectable effect (MDE) in online experiments by 42%, and significantly improves statistical power and metric reliability. Our approach establishes a new industrial-grade paradigm for search evaluation—efficient, scalable, and high-fidelity.
📝 Abstract
Relevance evaluation plays a crucial role in personalized search systems to ensure that search results align with a user's queries and intent. While human annotation is the traditional method for relevance evaluation, its high cost and long turnaround time limit its scalability. In this work, we present our approach at Pinterest Search to automate relevance evaluation for online experiments using fine-tuned LLMs. We rigorously validate the alignment between LLM-generated judgments and human annotations, demonstrating that LLMs can provide reliable relevance measurement for experiments while greatly improving the evaluation efficiency. Leveraging LLM-based labeling further unlocks the opportunities to expand the query set, optimize sampling design, and efficiently assess a wider range of search experiences at scale. This approach leads to higher-quality relevance metrics and significantly reduces the Minimum Detectable Effect (MDE) in online experiment measurements.