Guiding Retrieval using LLM-based Listwise Rankers

2025-01-15

AI Summary

To address the problem that listwise LLM re-rankers permanently miss highly relevant documents due to insufficient initial retrieval recall, this paper proposes the first adaptive retrieval framework tailored for listwise LLM re-ranking. Departing from the conventional assumption of independent document scoring, our method dynamically generates feedback from LLM re-ranking outputs and leverages it to guide multi-round retrieval in real time; final results are obtained via lightweight fusion of initial-retrieval and feedback-retrieved documents. Extensive experiments across diverse LLM re-rankers, first-stage retrievers, and feedback sources demonstrate improvements of up to 13.23% in nDCG@10 and 28.02% in recall—without increasing LLM inference cost. Our core contribution is the first integration of adaptive retrieval into the listwise LLM re-ranking paradigm, enabling closed-loop, synergistic optimization between retrieval and re-ranking.

Technology Category

Search and Optimization: Learning to SearchMachine Learning: Learning Preferences or RankingsData Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingUser Modeling, Personalization and Recommendation: Fairness-aware retrieval and rankingGraph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Language Models (LLMs) have shown strong promise as rerankers, especially in ``listwise'' settings where an LLM is prompted to rerank several search results at once. However, this ``cascading'' retrieve-and-rerank approach is limited by the bounded recall problem: relevant documents not retrieved initially are permanently excluded from the final ranking. Adaptive retrieval techniques address this problem, but do not work with listwise rerankers because they assume a document's score is computed independently from other documents. In this paper, we propose an adaptation of an existing adaptive retrieval method that supports the listwise setting and helps guide the retrieval process itself (thereby overcoming the bounded recall problem for LLM rerankers). Specifically, our proposed algorithm merges results both from the initial ranking and feedback documents provided by the most relevant documents seen up to that point. Through extensive experiments across diverse LLM rerankers, first stage retrievers, and feedback sources, we demonstrate that our method can improve nDCG@10 by up to 13.23% and recall by 28.02%--all while keeping the total number of LLM inferences constant and overheads due to the adaptive process minimal. The work opens the door to leveraging LLM-based search in settings where the initial pool of results is limited, e.g., by legacy systems, or by the cost of deploying a semantic first-stage.

Problem

• Large Language Models
• Search Result Relevance
• Adaptive Retrieval Techniques

Innovation

• Adaptive Method
• Large Language Models (LLMs)
• Search Performance Optimization