Don't `Well, Actually' Me Unless You Know What You're Talking About: Weak Presupposition Verification Degrades General QA Performance

2026-08-06

AI Summary

This study addresses a critical trade-off in current false-presupposition question answering (FPQA) approaches: while enhancing models’ ability to handle questions containing false presuppositions, these methods often degrade performance on standard factual questions. Through systematic evaluation across multiple large language models, the work reveals—for the first time—a significant negative correlation between FPQA robustness and general QA accuracy. The root cause is traced to weak fact-checking modules that erroneously reject valid presuppositions. Extensive experiments spanning diverse model families, scales, and benchmarks, combined with presupposition extraction and fact verification techniques, demonstrate that existing FPQA methods lack generalization in realistic settings. These findings offer crucial insights for designing future QA systems that simultaneously achieve high accuracy and robustness.

Abstract

False-presupposition QA (FPQA) tests LLMs on their ability to identify false presuppositions in questions and abstain or correct them rather than reinforcing false assumptions. The common approach reduces the task to prompting LLMs to extract presuppositions and fact checking each presupposition. While the performance on dedicated benchmarks keeps improving, evaluation largely focuses on questions with false presuppositions (FPQs) while ignoring the performance on ``normal'' questions (TPQs). Since many benchmarks over-represent FPQs compared to their natural occurrence, the result is that performance on these benchmarks doesn't reflect real-world QA performance. Through extensive experiments across various model families, sizes, and benchmarks, we show that methods that perform better on FPQs tend to perform worse on TPQs. Our analysis reveals this is the result of weak fact checking modules that reject also true presuppositions. We hope our findings will help guide future work toward FPQA methods that generalize well to realistic settings.

Problem

• False-presupposition QA
• presupposition verification
• general QA performance
• benchmark bias
• true presuppositions

Innovation

• False-presupposition QA
• presupposition verification
• fact checking
• generalization
• question answering