Reproducing LightMem: Naive RAG Is Just as Good for Memory Management

2026-07-31

AI Summary

This study addresses the core challenge in long-term dialogue systems of efficiently leveraging user history to avoid the high computational cost of processing full conversation logs while preserving essential information for response generation. The authors reproduce the LightMem approach and conduct a systematic comparison against a naive retrieval-augmented generation (RAG) baseline that retrieves from original dialogue turns. Through comprehensive experiments employing diverse retrievers, fixed memory storage, oracle evaluations, and matched retrieval depths, they quantitatively measure, for the first time, the amount of response-relevant information potentially lost during memory construction. Their findings reveal that LightMem does not universally outperform naive RAG; its advantage emerges only under stringent token budgets. Moreover, merely changing the retriever causes LightMem’s accuracy to vary between 58.1% and 75.5%, whereas naive RAG consistently achieves superior performance in most settings, particularly when token budgets are ample.

Abstract

Long-term conversational agents require access to information from earlier interactions, such as a user's preferences, past requests, or previously mentioned facts. Repeatedly providing the full dialogue history can be expensive as conversations grow, so many memory approaches instead transform past interactions into compact entries that can be retrieved when needed. LightMem is a recent lightweight memory-management approach that reports strong effectiveness while maintaining relatively low construction cost. However, it still relies on a separate constructed memory representation and is evaluated with only one retriever, leaving unclear how sensitive its results are to retriever choice and whether memory construction discards answer-relevant information. In this study, we reproduce LightMem and compare it with Naive RAG, which retrieves directly from raw user turns. We recover LightMem's main configuration trend, but find that retriever choice is a major source of performance variation: changing only the retriever over a fixed LightMem store shifts answer accuracy from 58.1% to 75.5%. Constructed memories also do not consistently outperform raw-turn retrieval. Naive RAG generally performs better at matched retrieval depths, whereas LightMem performs better mainly under tight answering-token budgets. Oracle evaluation further shows that memory construction removes some answer-relevant information. Overall, LightMem offers a context-efficiency trade-off rather than a general advantage over Naive RAG. Its value depends on the retriever and available token budget, motivating future work on retrieval, reranking, query formulation, and their interaction with raw and constructed memory representations.

Problem

• long-term conversational agents
• memory management
• retrieval-augmented generation
• memory construction
• retriever sensitivity

Innovation

• memory management
• Naive RAG
• retriever sensitivity
• context efficiency
• memory construction