Building Reliable Long-Form Generation via Hallucination Rejection Sampling

2026-06-02

AI Summary

This work addresses the challenge of error accumulation in long-form text generation by large language models, where early hallucinations can propagate and undermine factual consistency. To mitigate this “snowball effect,” the authors propose Semantic-level Hallucination-aware Rejection Sampling (SHARS), a novel framework that introduces rejection sampling into hallucination control for long texts. During inference, SHARS constructs a hallucination detector based on semantic uncertainty and dynamically resamples and filters low-confidence paragraphs, enabling self-correction without reliance on external knowledge. Experimental results demonstrate that SHARS significantly reduces hallucination rates on standard evaluation benchmarks while preserving or even enhancing the informativeness of generated content, thereby effectively curbing the propagation of hallucinatory errors.

Abstract

Large language models (LLMs) have achieved remarkable progress in open-ended text generation, yet they remain prone to hallucinating incorrect or unsupported content, which undermines their reliability. This issue is exacerbated in long-form generation due to hallucination snowballing, a phenomenon where early errors propagate and compound into subsequent outputs. To address this challenge, we propose a novel inference-time hallucination mitigation framework, named Segment-wise HAllucination Rejection Sampling (SHARS), which uses an arbitrary hallucination detector to identify and reject hallucinated segments during generation and resample until faithful content is produced. By retaining only confident information and building subsequent generations upon it, the framework mitigates hallucination accumulation and enhances factual consistency. To instantiate this framework, we adopt semantic uncertainty as the detector and introduce several vital modifications to address its limitations and better adapt it to long-form text. Our method enables models to self-correct hallucinations without requiring external resources such as web search or knowledge bases, while remaining compatible with them for future extensions. Empirical evaluations on standardized hallucination benchmarks demonstrate that our method substantially reduces hallucinations in long-form generation while preserving or even improving the informativeness of generation. Code is available at: https://github.com/TreeLLi/hallucination-rejection-sampling.

Problem

• hallucination
• long-form generation
• factual consistency
• reliability
• error propagation

Innovation

• hallucination rejection sampling
• long-form generation
• semantic uncertainty
• self-correction
• factual consistency