When Better Gets Worse: Improvement Fidelity for Self-Improving Agents in Adaptive Worlds

2026-09-26

AI Summary

This study addresses the failure of self-improvement decisions by agent evaluators in adaptive environments due to dynamic environmental changes. It introduces the concept of "improvement fidelity," demonstrating that global accuracy cannot guarantee update-level correctness. Methodologically, this work defines an improvement fidelity metric and develops PIVOT-KG, a paired decision-aware validation framework, which incorporates a regret minimization strategy to optimize the allocation of scarce evaluation resources. Experimental results across multiple game environments reveal significant non-overlap among optimal solution sets. The proposed approach reduces selection regret to 0.0055, effectively enhancing the reliability of agent self-improvement and its real-world post-deployment benefits.

Abstract

Self-improving agents increasingly rely on proxy verifiers to choose policy updates, yet deployment can change the world in which those updates are evaluated. An update that looks better to the verifier can therefore become worse after deployment even when the verifier ranks policies well overall. We formalize this gap as Improvement Fidelity, which asks whether proxy improvements preserve the sign and ordering of deployment improvements over the updates an improvement process actually proposes. We show that global policy accuracy need not guarantee update fidelity: operator shift and deployment response can create update-level errors, while candidate margins determine whether those errors change the replacement decision. We introduce PIVOT-KG, a paired, decision-aware validator that allocates scarce high-fidelity evaluation according to the expected reduction in selection regret per unit cost. Across 90 held-out roots in Leduc, Kuhn, and Melting Pot, proxy and deployment optimal sets are disjoint in 51 cases. In an eight-candidate HighwayEnv stress test, PIVOT-KG reduces mean improvement-selection regret from 0.0435 under the exact Uniform validation rule to 0.0055 at the primary budget. Together, these results show why reliable self-improvement should evaluate proposed improvements in the worlds they induce, while providing a practical rule for allocating scarce deployment evidence when it can affect the replacement decision.

Problem

• Self-improving agents
• Improvement fidelity
• Proxy verifiers
• Deployment response
• Selection regret

Innovation

• Improvement Fidelity
• PIVOT-KG
• Self-Improving Agents
• Proxy Verifiers
• Selection Regret