🤖 AI Summary
This work addresses the entanglement of reranking behavior with retrieval quality in existing evaluation paradigms, which hinders isolated analysis of reranking strategies themselves. To disentangle these factors, the authors propose a model-agnostic, controlled diagnostic framework that constructs fixed evidence pools—each strictly containing eight documents—via clustering on the Multi-News dataset, thereby standardizing inputs to isolate the reranking process. Using BM25 and MMR as interpretable baselines, they systematically evaluate diverse rerankers across 345 clusters. Under conditions that eliminate retrieval variance, the study reveals intrinsic differences among large language model (LLM)-based rerankers in terms of diversity and lexical coverage: some implicitly enhance diversity under high-budget settings while others introduce redundancy; under low-budget constraints, most LLM rerankers consistently underperform baselines and significantly deviate from ideal coverage patterns.
📝 Abstract
Standard reranking evaluations study how a reranker orders candidates returned by an upstream retriever. This setup couples ranking behavior with retrieval quality, so differences in output cannot be attributed to the ranking policy alone. We introduce a controlled diagnostic that isolates reranking by using Multi-News clusters as fixed evidence pools. We limit each pool to exactly eight documents and pass identical inputs to all rankers. Within this setup, BM25 and MMR serve as interpretable reference points for lexical matching and diversity optimization. Across 345 clusters, we find that redundancy patterns vary by model: one LLM implicitly diversifies at larger selection budgets, while another increases redundancy. In contrast, LLMs underperform on lexical coverage at small selection budgets. As a result, LLM rankings diverge substantially from both baselines rather than consistently approximating either strategy. By eliminating retrieval variance, we can attribute these differences directly to the ranking policy. This diagnostic is model-agnostic and applicable to any ranker, including open source systems and proprietary APIs.