Consistency-Guided Decoding with Proof-Driven Disambiguation for Three-Way Logical Question Answering

2026-03-12

AI Summary

This work addresses the issues of negation inconsistency and overuse of the “Unknown” label in large language models when performing ternary logical question answering. To this end, the authors propose CGD-PD, a training-free, test-time decoding framework that formulates the task as a compositional reasoning problem. CGD-PD integrates neural ternary classification with symbolic logical constraints—enforced via a negation consistency projection—and targeted binary entailment probes to ensure that during inference, each hypothesis and its negation adhere to a deterministic logical mapping. Experimental results on the FOLIO dataset demonstrate substantial performance gains: accuracy improves by 4.4 points for GPT-5.2 and 6.8 points for Claude Sonnet, while significantly reducing spurious “Unknown” predictions and thereby enhancing the model’s logical reliability.

Abstract

Three-way logical question answering (QA) assigns $True/False/Unknown$ to a hypothesis $H$ given a premise set $S$. While modern large language models (LLMs) can be accurate on isolated examples, we identify two recurring failure modes in 3-way logic QA: (i) negation inconsistency, where answers to $H$ and $\neg H$ violate the deterministic label mapping, and (ii) epistemic $Unknown$, where the model predicts $Unknown$ due to uncertainty or instability even when $S$ entails one side. We present CGD-PD, a lightweight test-time layer that (a) queries a single 3-way classifier on both $H$ and a mechanically negated form of $H$, (b) projects the pair onto a negation-consistent decision when possible, and (c) invokes a proof-driven disambiguation step that uses targeted binary entailment probes to selectively resolve $Unknown$ outcomes, requiring only an average of 4-5 model calls. On the FOLIO benchmark's first-order-logic fields, CGD-PD yields consistent gains across frontier LLMs, with relative improvements in accuracy of up to 16% over the base model, while also reducing $Unknown$ predictions.

Problem

• three-way logical question answering
• negation inconsistency
• epistemic Unknown
• compositional inference
• large language models

Innovation

• compositional inference
• negation consistency
• three-way logical QA
• training-free decoding
• epistemic abstention