🤖 AI Summary
This work addresses the lack of auditability and reproducibility in existing autonomous improvement loops, where failures often go unrecorded and repair decisions are decoupled from agent history. To remedy this, the authors propose ActiveGraph, an event-sourcing–based runtime that embeds the entire improvement process within the agent’s historical trace, ensuring end-to-end auditability. The approach incorporates a retention-set gating mechanism, failure-mode–aware routing, and prompt probes to diagnose failures, generate repairs in typed pipeline nodes, and validate them through static checks, sandboxed execution, in-sample evaluation, and retention-set verification before promotion. Evaluated on LongMemEval-S, the method yields significant accuracy gains in four out of five retention partitions (+0.05 to +0.10), with a modest +0.01 improvement in the fifth, indicating that the primary bottleneck lies in evidence integration rather than retrieval.
📝 Abstract
Autonomous improvement loops are hard to trust because the improvement process is usually external scaffolding bolted onto the agent: failures go unlogged, diagnoses cannot be replayed, and promote-or-discard decisions land in a side database rather than the agent's own history. We show that an event-sourced agent runtime removes that friction and turns controlled improvement into a first-class workflow. When the agent's state is a deterministic projection of an append-only event log, failures are recorded, a run replays exactly from its log, candidate patches scope to typed pipeline seams, gates are auditable, and every promotion or discard is itself an event. We demonstrate this with Regimes, a loop on the ActiveGraph runtime that diagnoses failed evaluations, proposes a repair at a pipeline point, and promotes it only after static checks, sandbox execution, in-sample evaluation, and held-out validation. The loop is target-agnostic: the same control flow runs against different tasks through a common interface. On LongMemEval-S the dominant failure is not retrieval but reconciliation: the evidence is already in the assembled context, yet the reader answers incorrectly. Across five seeded held-out splits, Regimes discovers reader-prompt repairs that improve final held-out accuracy by +0.05 to +0.10 in four splits and +0.01 in one over-promotion split; two splits are individually significant (seed 5 unadjusted for its sequential promotion structure), and the pooled count is descriptive only, since the splits share one 500-question pool. The durable contributions are ActiveGraph as an auditable substrate that makes controlled improvement loops tractable, the held-out-gated loop it supports, the failure-regime taxonomy routing each failure to a pipeline location (whose marginal value over an unrouted baseline is the primary open question), and the prompt-as-discovery-probe hypothesis.