Signed Compression Progress on a Sealed Audit is Goodhart-Resistant

2026-06-09

AI Summary

This work addresses the problem of reward hacking—manifesting as Goodhart’s law failures—in reinforcement learning agents driven by intrinsic rewards. To mitigate this, the authors propose signed compressed progress under a sealed audit mechanism as an intrinsic reward, ensuring that cumulative reward strictly corresponds to genuine performance gains on a fixed audit set. They formally define and prove the anti-Goodhart property of this reward scheme, characterize its failure boundary, and provide theoretical guarantees on bias control under finite auditing. Their analysis establishes that the reward cannot be exploited indefinitely; empirical results further demonstrate that audit bias decays at a rate of $n^{-0.527}$, and that the method effectively resists score-clipping exploits, data stream leakage, and “noisy-TV” attacks.

Abstract

Compression progress is a long-standing proposal for intrinsic motivation: reward an agent when its world model becomes better at predicting or compressing experience. The folk claim is that this reward is "credible" because it is paid only for learning. We make this precise and prove it. If intrinsic reward is the signed decrease of a fixed sealed-audit loss, r_t = E(theta_{t-1}) - E(theta_t), then cumulative reward telescopes exactly to endpoint audit improvement, so no policy can push reward up indefinitely while true audit performance stagnates or degrades. For finite audit panels the same result holds with a sharp false-positive budget: cumulative empirical reward is at most true audit improvement plus 2 Delta_n(F, delta), the uniform audit deviation of the model class. This is horizon-free: adaptivity over time costs nothing once the sealed panel uniformly controls the class. The theorem also identifies the failure modes: the guarantee disappears if progress is clipped, scored on the agent's own stream, exposed to a high-capacity model on a reusable panel, or applied to a neural class that makes Delta_n vacuous. We give a Lean 4 mechanization of the structural core (telescoping, the finite-audit bound, finite Gibbs, and the entropy floor) and an experiment suite on ARC-TGI grid-transformation generators with adaptive holdout attacks. Experiments confirm the theory: finite-audit deviation scales as n^{-0.527}; signed progress resists clip-farming, stream leakage, and noisy-TV curiosity; naive reusable audits are exploitable by black-box scalar feedback, while standard release defenses keep the attack below the 2 Delta_n threshold. Signed compression progress on a sealed audit is an accounting signal of genuine improvement.

Problem

• intrinsic motivation
• compression progress
• Goodhart resistance
• sealed audit
• reward manipulation

Innovation

• signed compression progress
• sealed audit
• Goodhart-resistance
• uniform convergence
• intrinsic motivation