RoMeRL: Balancing Feedback Coverage and the Memory-Reward Trap in Self-Evolving Agent Memory via Reduced-Order Utility States

2026-08-03

AI Summary

This work addresses two critical challenges in memory systems for self-evolving large language model (LLM) agents: sparse feedback and the entanglement between memory and reward signals. Specifically, trajectory expansion disperses utility signals, while co-retrieved memories lead to erroneous reward updates from irrelevant experiences. To mitigate these issues, the authors propose RoMeRL, a novel method that introduces a fixed-dimensional, reduced-order utility state representation for task memory. This representation factorizes memory states based on outcome polarity and dynamic memory factors, and continuously updates or replaces content within a bounded semantic coordinate set. The approach concentrates feedback signals, alleviates memory bloat, and reduces reward contamination. Theoretical analysis shows enhanced average feedback per coordinate and suppressed steady-state occupancy of incorrect coordinates. Experiments on ALFWorld and LifelongAgentBench demonstrate substantial performance gains, an 80.0% reduction in Cold-Q ratio, a 6× increase in feedback density, 84.4% less memory usage, and 21.1% fewer LLM invocations.

Abstract

Learning-based memory systems for self-evolving LLM agents face two tightly coupled challenges. First, trajectory-indexed utilities grow with the interaction history, thereby dispersing limited feedback over an ever-expanding state space. Second, because trajectory-level rewards are jointly assigned to co-retrieved memories, irrelevant experiences may receive misleading utility updates and consequently enter the memory-reward trap. To address these challenges, we introduce Reduced-Order Memory Reinforcement Learning (RoMeRL), which represents the growing trajectory-indexed utility space using a fixed-dimensional per-task memory state factorized by outcome polarity and memory dynamics. RoMeRL incorporates new experiences through a fixed set of semantic coordinates whose contents are updated or replaced over time, thereby concentrating feedback over a bounded utility support. Theoretically, we show that this reduced-order parameterization increases the average feedback received by each utility coordinate and characterize the steady-state occupancy of erroneous coordinates under a generic coordinate-transition model. Empirically, across ALFWorld and LifelongAgentBench, RoMeRL improves task performance, reduces the Cold-Q ratio by 80.0%, increases feedback density by approximately 6.0 times, reduces the maintained memory size by 84.4%, and cuts LLM calls by 21.1%. These results show that reduced-order utility states support efficient self-evolving agent memory while limiting persistent reward contamination. Code is available at: https://github.com/YOUNG-fnxm/RoMeRL

Problem

• self-evolving agents
• memory-reward trap
• feedback sparsity
• trajectory-indexed utilities
• memory contamination

Innovation

• Reduced-Order Utility States
• Memory-Reward Trap
• Self-Evolving Agents
• Feedback Density
• Trajectory-Indexed Memory