MemHarness: Memory Is Reconstructed, Not Replayed

2026-07-30

AI Summary

This work addresses a key limitation in existing memory-augmented agents, which typically replay static experiences without accounting for semantic mismatches between retrieved memories and the current context, often leading to negative transfer. To overcome this, the paper proposes MemHarness, a novel framework that abandons the conventional replay paradigm in favor of a human-like memory reconstruction mechanism. Within this framework, a large language model agent critically reinterprets retrieved memories in light of the current state during decision-making, generating dynamic, context-adapted guidance signals. MemHarness integrates memory retrieval, critique, and reconstruction into a unified policy model trained end-to-end via GRPO. Experiments on ALFWorld and WebShop demonstrate substantial improvements over pure reinforcement learning and static memory baselines, with strong robustness in out-of-distribution scenarios and effective mitigation of negative transfer.

Abstract

Retrieving past experiences has become a common strategy to enhance large language model agents. However, most existing memory-augmented agents treat retrieved experiences as static records to be replayed verbatim, injecting them into the context regardless of whether they align with the agent's current situation. This ``replay'' paradigm ignores the gap between the abstract, general nature of stored experience and the concrete, ever-changing states encountered at decision time, frequently causing negative transfer. In contrast, humans rarely recall past experiences verbatim; instead, they reorganize and adapt retrieved memories to fit the present context. Inspired by this, we propose MemHarness, a framework that equips LLM agents to actively harness and reconstruct past experiences based on the present context. At each decision step, a unified policy model critiques and reconstructs the retrieved experience conditioned on the current state, producing context-grounded guidance before acting. This reconstructive ability emerges naturally through end-to-end training with GRPO. Experiments on ALFWorld and WebShop show that MemHarness substantially outperforms pure RL and static memory-augmented baselines, demonstrating strong robustness in out-of-distribution (OOD) scenarios. Furthermore, our analyses reveal that this reconstruction objective not only prevents negative transfer but also serves as latent guidance during training, fundamentally improving the agent's intrinsic reasoning capabilities.

Problem

• memory augmentation
• negative transfer
• experience replay
• context alignment
• large language model agents

Innovation

• memory reconstruction
• LLM agents
• context-grounded reasoning
• negative transfer mitigation
• GRPO training