Iterative Utility Judgment Framework via LLMs Inspired by Relevance in Philosophy

2024-06-17

arXiv.org

AI Summary

To address the input bandwidth limitations of large language models (LLMs) in retrieval-augmented generation (RAG) and the difficulty of filtering highly utility-bearing retrieval results, this paper proposes ITEM, an Iterative Utility judgment framework. Inspired by Schutz’s phenomenological “theme–interpretation–motive” triadic relevance theory, ITEM is the first to formalize this theory as a computable iterative optimization framework that dynamically coordinates utility assessment, relevance ranking, and answer generation. Its core innovations include: (1) an LLM-based multi-turn utility discrimination mechanism; (2) triadic relevance alignment modeling; and (3) an embedded closed-loop optimization paradigm integrated into the RAG pipeline. Experiments on TREC DL, WebAP, and Natural Questions (NQ) demonstrate that ITEM significantly improves utility judgment accuracy, nDCG@10 (+12.3%), and exact match (EM) rate (+9.7%).

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Knowledge Representation and Reasoning: Preferences

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Utility and topical relevance are critical measures in information retrieval (IR), reflecting system and user perspectives, respectively. While topical relevance has long been emphasized, utility is a higher standard of relevance and is more useful for facilitating downstream tasks, e.g., in Retrieval-Augmented Generation (RAG). When we incorporate utility judgments into RAG, we realize that the topical relevance, utility, and answering in RAG are closely related to the three types of relevance that Schutz discussed from a philosophical perspective. They are topical relevance, interpretational relevance, and motivational relevance, respectively. Inspired by the dynamic iterations of the three types of relevance, we propose an Iterative utiliTy judgmEnt fraMework (ITEM) to promote each step of the cycle of RAG. We conducted extensive experiments on multi-grade passage retrieval and factoid question-answering datasets (i.e., TREC DL, WebAP, and NQ). Experimental results demonstrate significant improvements in utility judgments, ranking of topical relevance, and answer generation upon representative baselines, including multiple single-shot utility judging approaches. Our code and benchmark can be found at https://anonymous.4open.science/r/ITEM-B486/.

Problem

• Improving utility judgment in RAG systems
• Enhancing information prioritization for LLM inputs
• Aligning retrieval components with philosophical relevance concepts

Innovation

• Iterative utility judgment framework
• LLM-based utility prioritization
• Schutz's philosophical relevance integration