🤖 AI Summary
Existing evidence construction systems are difficult to diagnose across paradigms due to inconsistent corpora and evaluation metrics. This work proposes AuthTrace, a unified diagnostic benchmark built upon a single-author, topic-dense corpus, comprising 2,099 annotated instances. For the first time, it enables consistent evaluation of mainstream evidence construction paradigms under identical corpora and query sets, introducing fan-in gradient as a core diagnostic dimension. Through exhaustive gold-standard evidence annotation and multi-system comparative experiments, the study identifies evidence recall as a strong predictor of answer quality (r = 0.96), reveals distinct performance degradation patterns across paradigms under high fan-in conditions, and demonstrates the general ineffectiveness of full-context prompting.
📝 Abstract
Evidence construction systems--chunk retrieval, agent memory, knowledge-graph traversal, and thematic indexing--are evaluated on separate benchmarks with incompatible corpora and metrics, making cross-paradigm diagnosis impossible. We introduce AuthTrace, the first diagnostic benchmark that places all major paradigms on a single corpus and query set by exploiting the dual nature of single-author collections. Built on thematically dense corpora where all texts share style, topic, and vocabulary, AuthTrace provides 2,099 instances with exhaustive gold evidence and a fan-in gradient as the primary diagnostic axis. Comparing eight systems across two QA models, we find that (1) evidence recall--not precision--is the dominant predictor of answer quality (r = 0.96); (2) fan-in exposes paradigm-specific collapse patterns, with flat retrieval degrading 3x faster than structured-evidence systems; and (3) full-context prompting fails uniformly, establishing evidence construction as a necessary capacity beyond raw corpus exposure.