Reflection or Re-Generation? Why LLM Revision Fails Where Human Revision Succeeds

2026-07-30

AI Summary

It remains unclear whether large language models (LLMs) possess genuine error-correction capabilities during reflection and revision. This work proposes a Human–LLM Reflection Framework (HRF) to systematically analyze revision behaviors through controlled two-round editing experiments across self-evaluation, peer evaluation, and cross-agent settings. Leveraging information-theoretic metrics (reduction in cross-entropy), oracle-guided diagnostics, and randomized reordering baselines, the study reveals two failure modes of LLM reflection: near-zero information gain in objective tasks and even negative gain in subjective ones, indicating that LLM revisions amount to conditional regenerations rather than error-driven corrections. In contrast, humans consistently achieve positive information gain across task types. These findings suggest that without external informational input, LLMs struggle to reduce uncertainty and may even degrade high-quality initial outputs.

Abstract

Reflection, the ability to revisit and revise prior reasoning, is central to how humans improve their answers. Large language models (LLMs) are increasingly prompted to "reflect," yet whether this resembles human revision remains unclear. We introduce the Human-LLM Reflection Framework (HRF), a controlled two-pass protocol comparing human and LLM revision under identical conditions across self-, peer-, and cross-agent settings. Using an information-theoretic analysis based on per-iteration cross-entropy reduction, we find two failure modes of LLM reflection. On objective tasks with finite answer spaces, reflection yields near-zero information gain (Delta I approx 0), behaving as neutral re-generation indistinguishable from re-sampling. On subjective tasks, it yields significant negative gain (Delta I < 0), moving predictions away from the target. Human revision, by contrast, yields positive gain in both settings. Cross-agent experiments localize the failure to the revision step, not input quality: LLMs degrade even high-quality human responses. Diagnostic analyses (revision conditioned on first-pass correctness, and oracle-guided revision against a random-reshuffle baseline) show that which sub-step dominates varies by task and by model rather than reducing to a single mechanism: self-error detection is present on objective multiple-choice tasks but weak on subjective ones, and recovery under an oracle error signal exceeds the baseline for some models and falls below it for others. The unifying account is structural: without external information, self-conditioned revision cannot reduce uncertainty about the target, so LLM reflection is better understood as conditioned re-generation than as genuine error-driven revision.

Problem

• reflection
• large language models
• revision
• information gain
• human reasoning

Innovation

• reflection
• large language models
• information-theoretic analysis
• revision failure
• cross-agent evaluation