LazyMem: Retrieve Broadly, Construct Selectively for Efficient Long-Term Agent Memory

2026-07-17

AI Summary

This work addresses the limitations of large language model agents in long-term memory retention, which are constrained by finite context windows and the trade-off between recall and noise in conventional retrieval methods. The authors propose LazyMem, a framework that defers all memory construction to query time. Within a broad retrieval candidate pool, a lightweight model jointly optimizes relevance identification and faithful compression across overlapping parallel windows, selectively preserving critical information. This approach avoids information loss from write-time compression and enables efficient parallel processing. Experiments show that LazyMem-4B achieves an LLM-judge accuracy of 0.85 on LongMemEval using only 213 memory tokens—reducing token consumption by 21× compared to the strongest non-oracle baseline—and demonstrates strong zero-shot generalization with lower latency on LoCoMo.

Abstract

Long-term memory lets LLM agents reuse past interactions, but raw dialogue histories are verbose and information-sparse. Retrieving broadly improves evidence coverage yet overwhelms downstream reasoning with noise; compressing at write time reduces noise but irreversibly discards details the future query may need. We introduce LazyMem, which sidesteps this dilemma by deferring all memory construction to query time. A lightweight 4B model processes the retrieved candidate pool in overlapping parallel windows, selectively retaining and compressing only query-relevant content. The model is trained through supervised fine-tuning followed by group-based reinforcement learning with a format-gated composite reward that combines a rule-based action signal measuring selection accuracy with an LLM-judged quality signal measuring source faithfulness and query utility. On the LongMemEval benchmark, LazyMem-4B achieves an LLM-judge accuracy of 0.85 with only 213 memory tokens, 68.7$\times$ fewer than retrieval-only, and generalizes to LoCoMo (0.68) without target-domain training, while reducing mean latency over the prior query-time baseline. The 32B variant reaches 0.93, surpassing oracle-context references on aggregation-heavy question types. The code associated with this work is publicly available at https://github.com/allacnobug/LazyMem.

Problem

• long-term memory
• retrieval
• context window
• memory compression
• LLM agents

Innovation

• LazyMem
• query-time memory construction
• selective compression
• long-term agent memory
• reinforcement learning