Organize then Retrieve: Hierarchical Memory Navigation for Efficient Agents

2026-06-10

AI Summary

This work addresses the challenge of context inflation in large language model (LLM) agents during long-horizon tasks, which stems from their stateless nature and leads to degraded reasoning quality and increased computational overhead. Existing memory mechanisms struggle to balance information completeness with retrieval efficiency. To overcome this, we propose HORMA, a framework featuring a file-system-inspired hierarchical memory architecture that organizes experiences through bidirectional links between summarized entities and original trajectories. HORMA incorporates a lightweight reinforcement learning–driven navigation agent that distinguishes between information absence and contextual misdirection, dynamically refining the memory structure. By integrating structured memory construction with goal-oriented retrieval of minimally sufficient context, HORMA achieves substantial performance gains on ALFWorld, LoCoMo, and LongMemEval benchmarks, delivering superior efficiency–accuracy trade-offs—using as few as 22.17% of the tokens required by baseline methods—while generalizing effectively to unseen tasks.

Abstract

Large language model (LLM) agents struggle with long-horizon tasks due to their inherent statelessness, requiring all task-relevant information to be encoded in growing input contexts. The resulting degraded reasoning quality, increased inference cost, and higher latency necessitate efficient working memory mechanisms. However, existing approaches either rely on lossy compression or similarity-based retrieval, which often fail to capture temporal structure and causal dependencies required for multi-step agentic tasks. In this work, we present HORMA, a Hierarchical Organize-and-Retrieve Memory Agent that organizes experience into a file-system-like hierarchical structure, where summarized entities are linked to the corresponding raw trajectories, enabling efficient access without losing detailed information. HORMA decomposes working memory into two stages: structured memory construction and navigation-based retrieval. The construction module iteratively refines how experiences are structured by distinguishing between failures caused by missing information and those caused by misleading or overloaded context. The navigation module retrieves task-relevant context by traversing the hierarchy using a lightweight agent trained with reinforcement learning to select minimal yet sufficient context, thereby reducing latency along the critical execution path. Across ALFWorld, LoCoMo, and LongMemEval, HORMA improves task performance under constrained context budgets while requiring at most 22.17% of the baseline token usage in long conversation tasks. Compared to existing methods, it consistently achieves better efficiency-performance trade-offs and generalizes effectively to unseen tasks.

Problem

• long-horizon tasks
• working memory
• temporal structure
• causal dependencies
• context efficiency

Innovation

• hierarchical memory
• memory organization
• navigation-based retrieval
• LLM agents
• context efficiency