Beyond Relevance-Centric Retrieval: Rubric-Oriented Document Set Selection and Ranking

2026-07-22

AI Summary

Current retrieval systems evaluate documents in isolation and rely on nDCG for aggregation, thereby overlooking inter-document relationships such as redundancy, conflict, and complementarity, which impedes accurate assessment of the overall quality of document sets. To address this limitation, this work proposes the first set-level evaluation–diagnosis–optimization closed-loop framework. It introduces SetwiseEvalKit, a three-tier, nine-dimension benchmark for comprehensive setwise evaluation, and Rubric4Setwise, a training-free method that translates multidimensional scoring rubrics into signals for set selection and ranking. The proposed approach achieves state-of-the-art downstream generation performance in both short- and long-text scenarios, attaining optimal results with fewer retrieved documents and retrieval rounds.

Abstract

As large language models and AI agents become the primary consumers of search results, document set quality determines the upper bound of downstream generation. Yet existing evaluation systems remain confined to scoring documents independently and aggregating via nDCG, ignoring inter-document interactions (redundancy, conflict, complementarity) and unable to answer what makes one document set better than another. To address these issues, we propose a complete evaluate-diagnose-optimize framework. We design SetwiseEvalKit, a three-level, nine-dimension document set evaluation benchmark covering both short-form and long-form scenarios, comprising approximately 28K high-quality evaluation rubrics. We systematically evaluate 12 rerankers: even the best method achieves no more than 45% coverage, cross-document coordination dimensions are universally weak, and no single method maintains top performance across both settings. Building on this, we propose Rubric4Setwise, a training-free method that converts rubric-based evaluation criteria into document set selection signals, achieving the best downstream generation performance with fewer documents and search rounds. It is the only method that maintains state-of-the-art results across both scenarios, validating the effectiveness of closing the loop from evaluation to optimization.

Problem

• document set evaluation
• inter-document interactions
• relevance-centric retrieval
• rubric-based assessment
• setwise ranking

Innovation

• rubric-oriented retrieval
• document set evaluation
• SetwiseEvalKit
• Rubric4Setwise
• cross-document coordination