From Isolated Conversations to Hierarchical Schemas: Dynamic Tree Memory Representation for LLMs

2024-10-17

arXiv.org

AI Summary

Large language models (LLMs) suffer from inefficient long-term memory management, hindering complex reasoning and extended interactive contexts. To address this, we propose MemTree—a dynamic, growable tree-structured memory architecture that hierarchically organizes textual snippets, semantic embeddings, and abstraction levels to enable context-aware, incremental memory updating and retrieval. Our key contributions are: (1) a cognitive-inspired schema mimicking human conceptual organization, supporting semantics-driven node splitting and merging—departing from conventional flat memory paradigms; (2) an integrated mechanism combining semantic similarity matching, hierarchical tree construction, dynamic node aggregation, and tree-structured reasoning enhancement. Evaluated on multi-turn dialogue understanding and long-document question answering benchmarks, MemTree consistently outperforms strong baselines, achieving 12.6%–18.3% absolute gains on memory-structure-dependent tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Recent advancements in large language models have significantly improved their context windows, yet challenges in effective long-term memory management remain. We introduce MemTree, an algorithm that leverages a dynamic, tree-structured memory representation to optimize the organization, retrieval, and integration of information, akin to human cognitive schemas. MemTree organizes memory hierarchically, with each node encapsulating aggregated textual content, corresponding semantic embeddings, and varying abstraction levels across the tree's depths. Our algorithm dynamically adapts this memory structure by computing and comparing semantic embeddings of new and existing information to enrich the model's context-awareness. This approach allows MemTree to handle complex reasoning and extended interactions more effectively than traditional memory augmentation methods, which often rely on flat lookup tables. Evaluations on benchmarks for multi-turn dialogue understanding and document question answering show that MemTree significantly enhances performance in scenarios that demand structured memory management.

Problem

• Optimizes long-term memory management in large language models.
• Enhances context-awareness through dynamic, hierarchical memory organization.
• Improves performance in multi-turn dialogue and document question answering.

Innovation

• Dynamic tree-structured memory representation
• Hierarchical organization with semantic embeddings
• Adaptive memory enrichment via semantic comparison