Rethinking Faithfulness in LLMs: A Pairwise Context-Sensitive Perspective

2026-10-06

AI Summary

This study addresses the limitation of existing faithfulness evaluations that overlook models’ ability to dynamically answer or abstain based on whether the context provides support. We formally define faithfulness from a paired, context-sensitive perspective and introduce PFaithBench, a benchmark constructed with supporting and non-supporting paired contexts to empirically evaluate 39 models. Our results reveal a pronounced over-answering tendency in current models and demonstrate that improperly constructed training data readily induces shortcut learning. Furthermore, we quantitatively characterize the trade-off between answering and abstention data. This work establishes a new paradigm for understanding and enhancing the contextual faithfulness of large language models.

Abstract

Large language models (LLMs) are expected to answer questions faithfully based on the provided context, abstaining when the context information is insufficient to answer the questions. Existing faithfulness evaluations typically assess each question-context instance in isolation; however, such instance-level evaluation fails to capture a fundamental requirement of faithful behavior: the ability to adapt model responses to changes in available contexts. In particular, a model should provide correct answers when sufficient evidence is present and abstain when it is not. In this work, we propose a Pairwise Faithfulness Benchmark (PFaithBench) that evaluates whether a model can switch between answering and abstaining for the same question under supporting versus non-supporting contexts. Our evaluations across thirty-nine models with seven model families demonstrate that faithfulness fundamentally involves a trade-off between answering and abstaining, and that most current models exhibit a strong bias toward answering, with most faithfulness errors arising from over-answering, i.e., models tend to fabricate a response even when the provided context is insufficient. We further conduct a series of studies on faithfulness training under different data constructions. Our results show that training outcomes are highly sensitive to the specific composition of answering and abstaining data. Constructing answering and abstaining data from mismatched sources can cause models to rely on dataset-specific shortcuts rather than actual context sufficiency. Moreover, increasing answer-supervised data improves answering performance but exacerbates over-answering, while increasing abstaining data reduces hallucination but leads to over-abstention. The code and data are released at https://github.com/tmlr-group/PFaithBench.

Problem

• Faithfulness
• Large Language Models
• Over-answering
• Abstention
• Context-sensitive evaluation

Innovation

• Pairwise Faithfulness
• Context-Sensitive Evaluation
• Over-answering Bias
• Abstention Trade-off
• Faithfulness Training