Hierarchical Memory for High-Efficiency Long-Term Reasoning in LLM Agents

2025-07-23

AI Summary

To address inefficient retrieval and weak contextual coherence arising from loosely organized long-term memory in LLM agents, this paper proposes a multi-level memory architecture grounded in semantic abstraction hierarchies. The architecture employs hierarchical memory encoding, position-indexed embedding, and layered routing for retrieval—enabling explicit semantic associations among memory vectors and layer-wise localization without global similarity computation. Compared to conventional flat memory structures, our approach significantly improves memory access efficiency and semantic consistency during long-horizon reasoning. Evaluated on five long-term dialogue tasks from the LoCoMo benchmark, it consistently outperforms five baseline methods, demonstrating superior effectiveness and generalization capability in complex, multi-step reasoning scenarios.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Long-term memory is one of the key factors influencing the reasoning capabilities of Large Language Model Agents (LLM Agents). Incorporating a memory mechanism that effectively integrates past interactions can significantly enhance decision-making and contextual coherence of LLM Agents. While recent works have made progress in memory storage and retrieval, such as encoding memory into dense vectors for similarity-based search or organizing knowledge in the form of graph, these approaches often fall short in structured memory organization and efficient retrieval. To address these limitations, we propose a Hierarchical Memory (H-MEM) architecture for LLM Agents that organizes and updates memory in a multi-level fashion based on the degree of semantic abstraction. Each memory vector is embedded with a positional index encoding pointing to its semantically related sub-memories in the next layer. During the reasoning phase, an index-based routing mechanism enables efficient, layer-by-layer retrieval without performing exhaustive similarity computations. We evaluate our method on five task settings from the LoCoMo dataset. Experimental results show that our approach consistently outperforms five baseline methods, demonstrating its effectiveness in long-term dialogue scenarios.

Problem

• Enhancing long-term reasoning in LLM Agents with hierarchical memory
• Improving structured memory organization and retrieval efficiency
• Addressing limitations in current memory mechanisms for LLM Agents

Innovation

• Hierarchical Memory organizes memory multi-level semantically
• Index-based routing enables efficient layer-by-layer retrieval
• Positional index encoding links related sub-memories