Wrong Answers Can Also Be Useful: PlausibleQA -- A Large-Scale QA Dataset with Answer Plausibility Scores

2025-02-22

AI Summary

Existing QA benchmarks evaluate only answer correctness, neglecting plausible yet incorrect candidates—leading to coarse-grained model evaluation and limited robustness analysis. To address this, we propose PlausibleQA, the first large-scale, fine-grained QA benchmark explicitly designed to assess answer plausibility. It comprises 10K questions, 100K candidate answers, and 900K human-annotated pairwise plausibility comparisons. Crucially, PlausibleQA introduces continuous plausibility scores and multi-dimensional attribution explanations—novel features enabling nuanced plausibility modeling. Built via rigorous crowdsourced collaborative annotation and empirically validated, the benchmark establishes a new paradigm for plausibility-aware multiple-choice QA modeling and robustness evaluation. Experiments demonstrate its superiority over baselines in evaluating model discrimination capability and assessing distractor quality improvements. All data and annotations are publicly released.

Technology Category

• Natural Language Processing: Question Answering
• Reasoning under Uncertainty: Relational Probabilistic Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Large Language Models (LLMs) are revolutionizing information retrieval, with chatbots becoming an important source for answering user queries. As by their design, LLMs prioritize generating correct answers, the value of highly plausible yet incorrect answers (candidate answers) tends to be overlooked. However, such answers can still prove useful, for example, they can play a crucial role in tasks like Multiple-Choice Question Answering (MCQA) and QA Robustness Assessment (QARA). Existing QA datasets primarily focus on correct answers without explicit consideration of the plausibility of other candidate answers, limiting opportunity for more nuanced evaluations of models. To address this gap, we introduce PlausibleQA, a large-scale dataset comprising 10,000 questions and 100,000 candidate answers, each annotated with plausibility scores and justifications for their selection. Additionally, the dataset includes 900,000 justifications for pairwise comparisons between candidate answers, further refining plausibility assessments. We evaluate PlausibleQA through human assessments and empirical experiments, demonstrating its utility in MCQA and QARA analysis. Our findings show that plausibility-aware approaches are effective for MCQA distractor generation and QARA. We release PlausibleQA as a resource for advancing QA research and enhancing LLM performance in distinguishing plausible distractors from correct answers.

Problem

• Evaluating plausibility of incorrect answers in QA
• Enhancing MCQA and QARA with plausible distractors
• Creating a dataset for nuanced QA model assessments

Innovation

• PlausibleQA dataset with plausibility scores
• Enhanced MCQA and QARA model evaluations
• Large-scale answer justification for plausibility