Reasoning Error from Known Fact: Step-Level Self-Consistency Group Relative Policy Optimization for LLM

2026-07-21

AI Summary

This work addresses the susceptibility of large language models to context-induced, hard-to-detect factual hallucinations in multi-step reasoning. Focusing specifically on context-sensitive, step-level hallucinations—a previously underexplored issue—the paper proposes Step-level Self-Consistency Group Relative Policy Optimization (SSC-GRPO). This method leverages multi-trajectory sampling to compute step-level self-consistency scores, which serve as fine-grained reward signals to guide policy optimization. By aligning model behavior with internally consistent reasoning paths at each step, SSC-GRPO significantly enhances the faithfulness of multi-step inference. Empirical evaluations demonstrate state-of-the-art performance on both mathematical reasoning benchmarks and hallucination detection leaderboards, effectively mitigating factual errors in complex reasoning tasks.

Abstract

With the rapid advancement of large language models (LLMs), modern systems not only possess strong foundational capabilities and extensive knowledge, but can also solve complex problems via long, multi-step reasoning. However, as reasoning traces become longer, LLMs may produce a substantial amount of hallucinated content during the reasoning process, which is often difficult to detect. In this work, we conduct a fine-grained analysis of hallucinations arising in LLM reasoning and find that the reasoning traces are particularly prone to Context-Sensitive Factual Hallucinations: cases where the model actually has the relevant knowledge, yet makes factual errors due to contextual interference during reasoning. To address this issue, we propose Step-level Self-Consistency Group Relative Policy Optimization (SSC-GRPO), which assigns step-level rewards to reasoning traces by computing self-consistency scores of individual steps across multiple rollouts. Compared with prior methods, SSC-GRPO achieves state-of-the-art performance on both mathematical reasoning benchmarks and hallucination leaderboards. Our results offer a new perspective for detecting and mitigating hallucinations in the reasoning process of large language models.

Problem

• hallucination
• reasoning
• large language models
• context-sensitive
• factual error

Innovation

• Self-Consistency
• Step-Level Reward
• Hallucination Mitigation
• Relative Policy Optimization
• Context-Sensitive Factual Hallucination