Towards Semi-Automatically Comparing Keyword-Based and Semantic Search Accuracy

2026-09-29

AI Summary

This study addresses the absence of quantitative comparison methods between keyword search and semantic chat-based search by proposing a semi-automated evaluation framework. The framework introduces a novel quantitative comparison paradigm tailored for heterogeneous output formats, integrating ranking accuracy and information completeness metrics with domain-specific equivalence class analysis. By employing techniques such as Retrieval-Augmented Generation (RAG) and the Mann-Whitney U test, it enables objective assessment of both retrieval paradigms. Validation through an industrial case study demonstrates that semantic search significantly outperforms keyword search. This research bridges the gap in cross-paradigm quantitative retrieval comparison, providing a scientific basis for system selection.

Abstract

The increasing importance of Information Retrieval (IR) in managing large datasets has highlighted significant limitations in traditional keyword-based search systems. Context-aware chat-based search methods, such as Retrieval Augmented Generation (RAG), have recently emerged, but their evaluation compared to keyword-based systems often relies on subjective user feedback. A rigorous, quantitative comparison between these paradigms remains lacking. This work introduces a novel, preliminary framework to quantitatively assess IR accuracy of search systems that produce different output formats, such as lists and messages. It focuses on two key aspects: the ranking accuracy for keyword-based systems and the completeness of retrieved information for semantic chat-based systems. Our approach enables semi-automatic comparisons of semantic and keyword-based methods using interchangeable equivalence classes tailored to domain-specific contexts (e.g., companies or problems). We validate the framework through an industrial case study, demonstrating statistically significant improvements in context-aware search over keyword-based methods, supported by analyses including the Mann-Whitney U-Test. With its adaptable design, the proposed framework provides a strong foundation for objectively assessing keyword-based and semantic chat-based search methods.

Problem

• Information Retrieval
• Keyword-based Search
• Semantic Search
• Retrieval Augmented Generation
• Quantitative Evaluation

Innovation

• Information Retrieval
• Retrieval Augmented Generation
• Semi-automatic Evaluation
• Semantic Search
• Equivalence Classes