SPECTRA: Synthetic IR Test Collections with Relevance Oracles and Controlled Distractor Diagnostics

2026-05-29

AI Summary

This work addresses the scarcity of scalable, low-cost, and controllable test collections for information retrieval systems, particularly in scenarios involving sensitive data or early-stage system design where human-annotated relevance judgments are unavailable. To this end, the authors propose SPECTRA, a novel framework that decouples latent topic structure, surface text generation, metadata control, query intent modeling, and deterministic relevance oracles, enabling fine-grained control over long-tail term distributions, distractor proportions, and relevance labels in synthetic corpora. Using a single-threaded Python prototype, SPECTRA efficiently generates a reproducible test collection comprising 60,000 documents and 9.61 million tokens at a rate of 12,000–14,000 documents per second. The framework demonstrates strong diagnostic utility: as distractor prevalence increases to 36%, BM25’s nDCG@10 drops from 1.00 to 0.43, effectively supporting stress-testing of retrieval systems.

Abstract

Scalable information retrieval testing needs corpora that are large enough to stress index construction, ranking latency, query routing, and evaluation tooling, yet human-judged test collections remain expensive and may be unavailable when documents are private or still under design. This paper introduces SPECTRA, a reproducible framework for generating synthetic text corpora and retrieval test collections through a separation of latent topical structure, surface text realization, metadata controls, query intent generation, and deterministic relevance oracles. The framework is intended as a diagnostic complement to Cranfield-style and TREC-style evaluation, not as a replacement for human assessment. A single-process Python prototype generated corpora up to 60,000 documents and 9.61 million tokens while preserving controllable long-tail vocabulary growth and producing graded relevance labels for 96 queries. In the local simulation study, generation remained close to linear at roughly 12K to 14K documents per second, estimated Zipf slopes stayed near 0.86 in absolute value, and increasing cross-topic distractor text reduced BM25 nDCG@10 from 1.00 at 2% distractors to 0.43 at 36% distractors. These results show that lightweight synthetic corpora can expose retrieval-system scaling and failure modes before costly collection construction begins.

Problem

• information retrieval
• test collections
• synthetic corpora
• relevance assessment
• evaluation

Innovation

• synthetic test collections
• relevance oracles
• controlled distractor diagnostics
• scalable IR evaluation
• deterministic relevance labeling