Toward a Theory of Hierarchical Memory for Language Agents

2026-03-23

AI Summary

Existing language agents lack a unified formal framework for systematically comparing hierarchical memory designs. This work proposes the first formal theory that models the entire hierarchical memory process—from raw data to multi-granularity representations and constrained context retrieval—through three operators: extraction (α), coarsening (C), and traversal (τ). The framework defines a spectrum of self-contained memory representations, reveals coupling constraints between coarsening and traversal strategies, and enables information extraction, multi-level clustered compression, and query- and budget-aware context traversal. Its generality and explanatory power are validated through application to 11 representative systems spanning document hierarchies, dialogue memory, and agent trajectories.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Modeling other Agents
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Many recent long-context and agentic systems address context-length limitations by adding hierarchical memory: they extract atomic units from raw data, build multi-level representatives by grouping and compression, and traverse this structure to retrieve content under a token budget. Despite recurring implementations, there is no shared formalism for comparing design choices. We propose a unifying theory in terms of three operators. Extraction ($α$) maps raw data to atomic information units; coarsening ($C = (π, ρ)$) partitions units and assigns a representative to each group; and traversal ($τ$) selects which units to include in context given a query and budget. We identify a self-sufficiency spectrum for the representative function $ρ$ and show how it constrains viable retrieval strategies (a coarsening-traversal coupling). Finally, we instantiate the decomposition on eleven existing systems spanning document hierarchies, conversational memory, and agent execution traces, showcasing its generality.

Problem

• hierarchical memory
• context-length limitation
• formalism
• language agents
• design comparison

Innovation

• hierarchical memory
• formalism
• coarsening-traversal coupling
• self-sufficiency spectrum
• language agents