🤖 AI Summary
This work addresses the challenge of evaluating whether large language models (LLMs) generate scientifically grounded answers with verifiable citation support in reading assistance tasks. To this end, the authors introduce a benchmark dataset comprising 6,211 question–answer pairs spanning eight disciplines and four question types, along with a novel evaluation framework that supports multi-segment valid citations and citation-based refusal mechanisms. A hybrid assessment approach combines a deterministic citation matcher with an LLM-based scorer to systematically evaluate both open- and closed-source models. The results demonstrate that citation-based metrics more effectively discriminate model performance than LLM-as-a-judge scores; notably, open-source models achieve citation accuracy comparable to the best closed-source counterparts while exhibiting 3–6 times lower inference latency.
📝 Abstract
Large language models are increasingly used to assist scientific reading, but existing evaluation methods often fail to detect whether answers are supported by verifiable citations. We introduce ResearchQA, a benchmark of 6,211 single-paper question-answer pairs from 494 open-access papers spanning eight domains and four question types: lookup, comprehension, multi-hop, and adversarial. ResearchQA is designed for citation-grounded evaluation: it permits multiple valid supporting passages for a claim and rewards grounded refusal when the source paper does not support an answer. We evaluate eight leading closed- and open-weight models in a citation-grounded chat-with-paper setting using a deterministic citation matcher and an LLM-based rubric evaluator. Citation-based metrics separate systems more clearly than LLM-evaluator scores: section coverage and citation accuracy vary substantially across models, while evaluator scores remain tightly compressed. We further find that open-weight models approach the best closed-model citation accuracy while achieving 3 to 6 times lower per-example latency. We release the benchmark, evaluation harness, and evaluator prompt.