Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory

2026-08-07

AI Summary

This work addresses the limited tool-use capability of small-scale language model agents, which stems from their difficulty in generating sufficient successful execution trajectories. To overcome this, the authors propose a training-free hierarchical memory distillation framework that systematically transfers structured knowledge from a large-model teacher to a small-model student. The framework encodes successful experiences into three types of memory: task-level workflows, subtask behavior examples, and function-calling specifications, and integrates both proactive injection prior to task execution and passive retrieval during inference. This approach achieves the first structured knowledge transfer tailored for small-agent models, yielding significant performance gains—average accuracy improvements of 27.2%, 11.2%, and 3.4% across three tool-use benchmarks—outperforming existing memory-augmentation methods, with the most pronounced gains observed in 4B-scale models.

Abstract

Memory systems have shown promise for improving agent performance, but their potential remains largely unexplored for small language models, which struggle to generate sufficient successful trajectories on their own. We propose Agent Memory Distillation (AMD), a training-free framework that transfers structured knowledge from a large teacher agent to a small student agent through hierarchical memory. AMD constructs three complementary memory types from successful teacher trajectories: Workflow memory encodes task-level strategies, Subtask memory provides concrete behavioral examples at an intermediate granularity, and Function memory captures per-function calling conventions and common pitfalls. Workflow and Subtask memories are injected proactively at the start of each task, while Function memory is retrieved reactively upon tool-calling errors. We evaluate AMD on three tool-use benchmarks using four student models (4B-8B parameters) with GPT-5-mini as the teacher, achieving average accuracy gains of 27.2%p, 11.2%p, and 3.4%p on AppWorld, BFCL V3, and ToolSandbox, while consistently outperforming existing memory-based baselines. Further analysis shows that Subtask memory contributes the largest gains, teacher effectiveness depends on both teacher capability and student compatibility, and 4B-sized students benefit most from AMD.

Problem

• small language models
• agent memory
• knowledge transfer
• tool-use tasks
• trajectory generation

Innovation

• Agent Memory Distillation
• Hierarchical Memory
• Memory Transfer
• Small Language Models
• Tool-Use Agents