Inference-Time Conformal Reasoning with Valid Factuality Control for Large Language Models

2026-06-07

AI Summary

This work addresses the prevalence of factual errors in multi-step reasoning by large language models, which often stem from structural dependencies and are inadequately mitigated by existing post-hoc calibration methods that lack dynamic intervention during generation. To overcome this limitation, the paper introduces In-Task Conformal Reasoning (ITCR), a novel framework that integrates conformal prediction directly into the reasoning graph generation process. ITCR aggregates node-level signals via a graph-level factual uncertainty function and dynamically halts generation based on nonconformity scores. This approach enables structure-aware, in-task control of factual consistency while providing theoretical coverage guarantees—surpassing the constraints of traditional post-hoc calibration. Empirical results demonstrate that ITCR achieves effective calibrated coverage across multiple datasets, and its calibrated reasoning graphs significantly outperform those produced by post-hoc pruning in downstream tasks.

Abstract

Large language models (LLMs) increasingly perform multi-step reasoning, where intermediate claims form implicit directed acyclic graphs whose node correctness is structurally conditioned on their ancestors. This makes factuality uncertainty structural, rather than a trivial accumulation of node-wise errors, and necessitates inference-time uncertainty quantification over the reasoning structure. While conformal prediction (CP) offers flexible user-specified factuality control, existing work remains post-hoc and cannot intervene during generation. To fill the gap between CP's flexibility and its post-hoc limitation, we propose an \emph{Inference-Time Conformal Reasoning (ITCR)} framework that integrates CP directly into reasoning graph generation. ITCR learns a structure-level factuality uncertainty function that aggregates claim-level factuality signals over reasoning graphs without complex modeling assumptions. We then design the non-conformity score based on graph-level factuality uncertainty and calibrate the conformal threshold to decide when to stop generation. We theoretically show such generation is nested, yielding valid coverage guarantees for factuality control. Experiments over multiple datasets and coverage objectives demonstrate empirically valid coverage. In downstream reasoning tasks, inference-time calibrated graphs yield more accurate generation than post-hoc pruned graphs.

Problem

• conformal prediction
• factuality control
• reasoning graphs
• inference-time intervention
• structural uncertainty

Innovation

• Conformal Prediction
• Inference-Time Reasoning
• Factuality Control
• Reasoning Graph
• Uncertainty Quantification