Distilling Agentic Systems: A Roadmap across Models, Artifacts, and Harnesses

2026-09-28

AI Summary

This study addresses the limitation of traditional knowledge distillation, which is confined to parameter imitation and fails to transfer the supra-parametric capabilities upon which agents rely, such as memory, tool use, and execution logic. We define agent distillation as the persistent transfer of task-solving knowledge and propose a taxonomic perspective based on knowledge retention loci, encompassing intra-model, artifact-based, execution framework, and cross-substrate dimensions. Furthermore, this work constructs a causal evaluation framework that disentangles transfer evidence from outcomes. By establishing a theoretical roadmap for multi-substrate knowledge transfer, this research lays the foundation for developing reliable, maintainable, and secure complex agent systems.

Abstract

Modern agents increasingly rely on memories, tools, and execution logic, so their competence extends beyond model parameters. This shift exposes a limitation of conventional knowledge distillation, which asks how a student model imitates a teacher model. We define Agent Distillation as the persistent transfer of task-solving knowledge from a teacher agent to a student agent. Our study organizes the field by where transferred knowledge is retained: within the model, as artifacts, through the execution harness, or across substrates. This perspective separates transfer evidence from its outcome and clarifies how knowledge moves between agent components. We develop an evaluation framework that relates retention to causal contribution and deployed utility. Together, these contributions establish a foundation for the reliable, maintainable, and safe development of increasingly complex agentic systems.

Problem

• Agent Distillation
• Knowledge Distillation
• Agentic Systems
• Knowledge Transfer

Innovation

• Agent Distillation
• Knowledge Transfer
• Evaluation Framework
• Agentic Systems
• Execution Harness