Learning on the Job: Continual Learning from Deployment Feedback for Frozen-Weights Agents

2026-07-24

AI Summary

This work addresses the challenge that AI agents with frozen weights after deployment struggle to learn continuously from experience, often failing on repeated tasks. The authors propose a continual learning mechanism leveraging external memory, which distills minimal feedback—either a single-bit outcome or natural language corrections—from each interaction into retrievable rules. Integrated with retrieval-augmented generation (RAG) and frozen large language models (e.g., Mistral Large, Claude Sonnet 5), this approach enables performance improvement without fine-tuning. The method demonstrates, for the first time, that extremely sparse feedback alone can drive sustained enhancement in frozen models and supports memory transfer across models. On the τ-bench banking tasks, it achieves success rates 1.6× (outcome-only feedback) and 2.6× (with corrections) higher than baseline, resolving 22 out of 84 tasks on which the baseline completely fails.

Abstract

AI agents encounter learning opportunities in every episode they run, and discard nearly all of them: the underlying models are frozen at deployment, so an agent that resolves a difficult request today starts from zero when it recurs tomorrow. Yet ordinary operation already produces feedback, in the form of outcome verdicts and after-the-fact corrections. We show that this feedback is a sufficient signal for continual learning when the frozen model is paired with an external memory that distils each episode into retrievable natural-language rules. On the banking domain of $τ$-bench, against a static-RAG control retrieving over the complete policy corpus, learning from the one-bit outcome verdict lifts single-trial success to 1.6$\times$ the baseline, and learning from corrections to 2.6$\times$, converting 22 of the 84 tasks the baseline never solves. The result spans the deployment spectrum, measured on Mistral Large, an open-weights model that organisations with data sovereignty requirements can self-host, and replicated on a frontier model, Claude Sonnet 5. The accumulated memory also transfers: each model, reading the store built by the other, rises above its own no-memory baseline. The harness, protocol, and data are released.

Problem

• continual learning
• frozen-weights agents
• deployment feedback
• external memory
• task recurrence

Innovation

• continual learning
• frozen-weights agents
• external memory
• deployment feedback
• retrieval-augmented generation