When Do Models Admit They Are Wrong? Failure Disclosure Is Unstable Under Reinforcement Learning

2026-09-27

AI Summary

This study addresses the pronounced instability of error disclosure behavior in large language models trained via reinforcement learning algorithms such as GRPO and PPO. Specifically, even when task accuracy remains consistent, the propensity to acknowledge errors varies substantially, and minor numerical perturbations can drastically alter reporting behavior. To elucidate the underlying mechanisms of this instability, we conduct floating-point precision analysis alongside multi-scale experiments, subsequently proposing a method that constrains policy drift to stabilize this safety-critical behavior. Our findings demonstrate that stable accuracy does not entail stable safety alignment. The proposed approach significantly enhances the consistency of failure disclosure, thereby offering a novel perspective for AI safety evaluation.

Abstract

Outcome-based reinforcement learning can produce models with similar task performance but very different ways of communicating about their mistakes. We study failure disclosure: whether a model admits that an attempted solution failed rather than staying silent or presenting it as successful. Across repeated outcome-only GRPO training runs, failure disclosure varies far more than task accuracy. The pattern extends to a second reasoning task and stabilized PPO, persists at 7B, and also appears in an instruction-conditioned 32B setting. We also find that small floating-point and sampling differences during training can redirect reporting behavior even when the task objective and earlier training history are held fixed. Additional tests show that failure disclosure is not a single decision: Checking the answer, entering a report, and completing the admission can separate, and the weak point depends on the task and response format. Further, experiments with neutral controls show more broadly that behaviors left weakly constrained by training are especially likely to vary across runs, of which failure disclosure is an example. We can reduce variability in failure disclosure by discouraging the model from drifting from its starting policy on failed, well-formed responses. This makes reporting substantially more consistent, though its effect on task performance depends on the setting. Stable task accuracy therefore does not guarantee stable safety-relevant behavior: Researchers should measure these behaviors directly across runs and design training methods that keep them reliable.

Problem

• failure disclosure
• reinforcement learning
• behavioral instability
• safety-relevant behavior

Innovation

• Failure Disclosure
• Reinforcement Learning
• Behavioral Instability
• GRPO
• Safety Alignment