LitTraceQA: A Benchmark for Multi-Stage Grounding and Verification in Scientific Question Answering

2026-08-07

AI Summary

Existing scientific question-answering systems struggle to reliably retrieve relevant papers from academic literature, locate fine-grained evidence, and generate faithful answers. To address this limitation, this work introduces a multi-stage scientific QA benchmark that jointly evaluates paper retrieval, evidence localization—including diverse scientific evidence such as tables, figures, and equations—and multi-format answer generation. The benchmark incorporates a structured evaluation pipeline to support verifiable question answering and releases a large-scale annotated dataset comprising 55 development examples and 4,978 questions grounded in 4,859 scholarly papers. This resource establishes standardized support for stage-wise performance assessment in scientific QA systems.

Abstract

Scientific literature is increasingly used as a knowledge source for language models, retrieval-augmented generation systems, and research assistants, but answering research questions from papers requires more than fluent generation. A reliable system must identify the relevant papers, locate the concrete evidence that supports the answer, and produce a response that is faithful to that evidence. We present LitTraceQA, a benchmark for literature-grounded question answering over scientific papers. Given a research question and a metadata pool of papers, a system must return three connected outputs: canonical paper identifiers, supporting evidence locations, and answers in one or more requested formats, including free-form text, multiple-choice answers, and structured tables. LitTraceQA targets evidence types common in scientific reading: tables, figures, text spans, equations or algorithms, and citation contexts. The public development split contains 55 examples, including 26 hidden-source single-paper questions and 29 multi-paper questions, and provides gold papers, evidence annotations, and answers for local validation. We also analyze a larger final annotation collection with 4,978 unique-question records over 4,859 unique gold papers. By evaluating paper retrieval, evidence grounding, and answer accuracy separately, LitTraceQA provides a testbed for scientific QA systems that produce verifiable answers rather than unsupported summaries.

Problem

• scientific question answering
• evidence grounding
• verifiable answers
• literature-based QA
• retrieval-augmented generation

Innovation

• scientific question answering
• evidence grounding
• retrieval-augmented generation
• verifiable QA
• multi-stage reasoning