Temporal Order Matters for Agentic Memory: Segment Trees for Long-Horizon Agents

2026-06-03

AI Summary

This work addresses the limitation of existing memory systems for long-horizon conversational agents, which predominantly rely on topical similarity while neglecting the temporal structure of events. To overcome this, the authors propose SegTreeMem—a novel memory architecture based on segment trees that, for the first time, integrates a strictly time-ordered segment tree into agent memory. This design enables online incremental updates at the right boundary to maintain a hierarchical temporal structure of dialogue history and supports retrieval by jointly leveraging local semantic matching and global temporal context. Experimental results demonstrate that SegTreeMem significantly outperforms baseline approaches—including flat, graph-based, and tree-based memory structures—across three long-horizon memory benchmarks and two large language models. Ablation studies further confirm that preserving temporal information is crucial for performance gains.

Abstract

Long-horizon conversational agents need to interact with users through evolving events, tasks, and goals. Such histories are naturally temporal, yet many existing memory systems organize information primarily by topical similarity and may ignore the order in which events occur. We introduce Segment Tree Memory, or SegTreeMem, a memory architecture that represents conversation history as a temporally ordered Segment Tree over utterances. SegTreeMem incrementally inserts new utterances through an online rightmost-frontier update rule, preserving chronological order while forming hierarchical memory segments. For retrieval, SegTreeMem propagates relevance scores through the tree to combine local semantic matching with hierarchical temporal context. Across three long-horizon memory benchmarks and two LLM backbones, SegTreeMem improves answer quality over flat retrieval, graph-structured memory, and tree-structured memory baselines. Additional temporal-order permutation analysis shows that the performance gain depends on preserving temporal order during memory construction, supporting the claim that temporal order is a key structure for agentic memory.

Problem

• temporal order
• agentic memory
• long-horizon agents
• conversation history
• memory systems

Innovation

• Segment Tree Memory
• temporal order
• long-horizon agents
• hierarchical memory
• online memory update