MemoryCPT: An End-to-End Agent Memory Framework for Cost-Performance Trade-off

2026-08-05

AI Summary

This work addresses the challenges of efficient retrieval and context compression faced by long-horizon large language model agents when processing extensive interaction histories, which often lead to high inference costs and performance bottlenecks. The authors propose MemoryCPT, the first end-to-end trainable memory framework that unifies offline memory construction with online query-conditioned context generation. By jointly optimizing Query-Agnostic Distillation (QAD) and Query-Aware Retrieval-summarization (QAR), MemoryCPT enables efficient memory management. The study introduces a novel Quality-per-Cost (QPC) metric and integrates reciprocal rank fusion (RRF), LoRA fine-tuning, and Group Relative Policy Optimization (GRPO). Evaluated on the LoCoMo and LongMemEval benchmarks, MemoryCPT significantly outperforms existing methods by improving response quality while controlling inference cost, with ablation studies confirming the effectiveness of each component.

Abstract

Long-horizon LLM agents require memory systems that recover useful evidence from large interaction histories without passing excessive context to downstream models. Existing memory pipelines often rely on hand-crafted heuristics and repeated LLM calls, which can introduce redundant context and high inference cost. We propose MemoryCPT, an end-to-end trainable agent memory pipeline that spans offline memory construction and online query-conditioned context generation. MemoryCPT consists of two stages: Query-agnostic Distillation (QAD), which distills a modular memory-construction pipeline into a compact model using explicit reasoning traces; and Query-aware Retrieval and Summarization (QAR), which combines reciprocal rank fusion (RRF) with a LoRA-based summarizer trained via Group Relative Policy Optimization (GRPO) under a cost-aware reward. We further introduce Quality per Cost (QPC) to quantify answer quality per unit inference cost. Experiments on LoCoMo and LongMemEval show that MemoryCPT improves the cost-performance trade-off over the evaluated baselines, while ablation and sensitivity analyses characterize the contributions of its components and the effects of key design choices.

Problem

• long-horizon LLM agents
• agent memory
• cost-performance trade-off
• memory systems
• inference cost

Innovation

• MemoryCPT
• cost-performance trade-off
• query-aware retrieval
• end-to-end trainable memory
• Group Relative Policy Optimization