Stop When Memory Suffices: Evidence-Conditioned Progressive Execution for LLM Agents

2026-08-02

AI Summary

This work addresses the challenge of balancing inference latency and response quality in long-horizon large language model agents by proposing the Router-Mem framework. The approach introduces a lightweight sufficiency router that, after low-overhead shared retrieval, dynamically decides—based on evidence conditions—whether to terminate reasoning early if evidence is sufficient or to reuse retrieved results to expand memory blocks for deeper analysis. Through evidence-level supervised training and rationale-conditioned representation distillation, the framework jointly optimizes response quality and computational efficiency. Evaluated on AMA-Bench and BEAM, Router-Mem achieves accuracies of 55.17% and 38.77%, respectively, while reducing average inference time by 27.3% and 25.5% compared to full-memory execution.

Abstract

The continued development of LLMs toward persistent and adaptive intelligence increasingly requires long-term memory mechanisms that preserve and reuse information across interactions. Existing memory systems either compress and structure histories for efficient access or perform deep research over broader trajectories. The former lowers online cost but may omit temporal, causal, or cross-step dependencies, while the latter improves evidence coverage at substantial latency and inference cost. This raises a key question: can a memory system achieve strong answer quality while maintaining low online latency? We introduce Router-Mem, an evidence-conditioned progressive execution framework for long-horizon agent memory. Router-Mem first applies a shared low-cost retrieval prefix to obtain evidence. A lightweight sufficiency router then predicts whether the context supports early termination, which enable a single-token decision at inference time. It is trained with evidence-level supervision and rationale-conditioned representation distillation. When evidence is insufficient, Router-Mem reuses retrieval hits to expand memory blocks and perform deeper analysis and aggregation. Experiments on AMA-Bench and BEAM show that Router-Mem achieves 55.17\% and 38.77\% score while reducing average inference time by 27.3\% and 25.5\% compared with full memory execution.

Problem

• long-term memory
• online latency
• answer quality
• LLM agents
• evidence sufficiency

Innovation

• evidence-conditioned execution
• progressive inference
• memory sufficiency routing
• retrieval reuse
• low-latency LLM agents