AR-Med: Automated Relevance Enhancement in Medical Search via LLM-Driven Information Augmentation

2025-12-03

AI Summary

Online medical search faces challenges including semantically complex user queries, inadequate understanding by conventional methods, and high deployment risks associated with large language models (LLMs)—notably factual hallucinations, lack of domain-specific medical expertise, and prohibitive inference costs. To address these, we propose AR-Med: a framework integrating retrieval-augmented generation (RAG) with lightweight knowledge distillation, grounded in authoritative medical knowledge bases to ensure factual grounding, substantially mitigate hallucinations, and reduce model footprint. We further introduce LocalQSMed, a multi-expert-annotated benchmark for localized query understanding in medicine, enabling consistent offline evaluation and online performance validation. Experiments demonstrate an offline accuracy of 93.2%, outperforming baselines by 24.1 percentage points; online deployment yields significant improvements in search relevance and user satisfaction. AR-Med has been successfully scaled across leading online healthcare platforms.

Technology Category

• Search and Optimization: Learning to Search
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: ML for personalized search and recommendations
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Accurate and reliable search on online healthcare platforms is critical for user safety and service efficacy. Traditional methods, however, often fail to comprehend complex and nuanced user queries, limiting their effectiveness. Large language models (LLMs) present a promising solution, offering powerful semantic understanding to bridge this gap. Despite their potential, deploying LLMs in this high-stakes domain is fraught with challenges, including factual hallucinations, specialized knowledge gaps, and high operational costs. To overcome these barriers, we introduce extbf{AR-Med}, a novel framework for extbf{A}utomated extbf{R}elevance assessment for extbf{Med}ical search that has been successfully deployed at scale on the Online Medical Delivery Platforms. AR-Med grounds LLM reasoning in verified medical knowledge through a retrieval-augmented approach, ensuring high accuracy and reliability. To enable efficient online service, we design a practical knowledge distillation scheme that compresses large teacher models into compact yet powerful student models. We also introduce LocalQSMed, a multi-expert annotated benchmark developed to guide model iteration and ensure strong alignment between offline and online performance. Extensive experiments show AR-Med achieves an offline accuracy of over 93%, a 24% absolute improvement over the original online system, and delivers significant gains in online relevance and user satisfaction. Our work presents a practical and scalable blueprint for developing trustworthy, LLM-powered systems in real-world healthcare applications.

Problem

• Enhance medical search relevance using LLMs for complex queries
• Mitigate LLM hallucinations and knowledge gaps in healthcare applications
• Balance accuracy with operational costs for scalable medical search systems

Innovation

• Retrieval-augmented LLM grounding for medical accuracy
• Knowledge distillation for efficient online deployment
• Multi-expert benchmark to align offline and online performance