CASE: Causal Alignment and Structural Enforcement for Improving Chain-of-Thought Faithfulness

2026-07-21

AI Summary

This work addresses the frequent lack of faithfulness in chain-of-thought (CoT) reasoning generated by large language models, where answers are often derived by bypassing the intended reasoning process. Framing CoT causally as Z→X→Y (instruction → reasoning chain → answer), the authors propose the CASE framework: during training, counterfactual data and a selective loss function strengthen the causal influence of the reasoning chain on the final answer; during inference, attention masking blocks the direct path from instruction to answer. Integrating causal alignment with structural constraints, CASE achieves an average 37% relative improvement in CoT faithfulness across three models and four benchmarks, significantly enhancing interpretability, reliability, and cross-dataset generalization while maintaining competitive accuracy.

Abstract

Chain-of-thought (CoT) reasoning is widely used to improve both the performance and interpretability of large language models (LLMs), yet the generated reasoning may not faithfully support the final answer. We study this problem from a causal perspective, where a faithful CoT process should follow the chain $Z\rightarrow X\rightarrow Y$, with $Z$, $X$, and $Y$ denoting the instruction, reasoning chain, and final answer, respectively. In this process, the instruction should affect the answer only through the reasoning chain. However, conventional autoregressive LLMs condition answer generation on both the instruction and the CoT, which still allows a direct instruction-to-answer shortcut. To address this issue, we propose CASE, a framework that combines training-time causal alignment and inference-time structural enforcement. During training, CASE builds counterfactual-CoT, biased-instruction, and empty-instruction datasets, and applies selective-loss fine-tuning to strengthen CoT-to-answer dependence while suppressing instruction shortcuts. During inference, CASE masks direct attention from instruction tokens to answer tokens, preventing the model from bypassing the generated CoT. We provide an information-theoretic analysis showing how these components promote faithful chains. Experiments on three models and four benchmarks show that CASE achieves a 37\% average per-setting relative improvement in overall CoT faithfulness over the strongest baselines, exhibits stronger cross-dataset faithfulness transfer, and maintains competitive average accuracy. Code is available at https://github.com/oddwang/CASE.

Problem

• Chain-of-Thought
• Faithfulness
• Causal Reasoning
• Language Models
• Instruction-to-Answer Shortcut

Innovation

• Causal Alignment
• Structural Enforcement
• Chain-of-Thought Faithfulness
• Selective-Loss Fine-tuning
• Attention Masking