Sample-Efficient Learning from Agent Experience

2026-07-23

AI Summary

This work addresses the challenge of enabling agents to efficiently internalize limited experience for continual performance improvement without incurring additional environmental interaction costs. The authors propose “experience distillation,” a method that integrates in-context learning with supervised fine-tuning to effectively consolidate knowledge from historical interactions directly into model parameters, eliminating the need for extra data collection. Experiments across 749 software engineering tasks and six text-based adventure games demonstrate that the approach retains at least 64.8% of the gains achieved through in-context learning, substantially outperforming standard supervised fine-tuning (which preserves only 3.8%) and reinforcement learning baselines—yielding over a 9.6× improvement in sample efficiency. This enables highly effective, low-interaction-cost continual learning.

Abstract

Real-world agent learning is often constrained by costly environment interactions, such as running time-consuming experiments or obtaining human feedback. In-context learning offers a highly sample-efficient way for agents to learn from their own interaction histories, but its gains disappear once that experience is removed from the context. Separately, context distillation provides a mechanism for internalizing contextual information into model weights. However, applying it to agents' interaction histories without sacrificing environment sample efficiency remains underexplored. We term this problem Experience Distillation and develop an implementation that requires no further environment interaction beyond the collected experience. Experiments on 749 curated software-engineering tasks and six text-adventure games show that it retains at least 64.8\% of the gains from in-context learning across both domains, whereas direct supervised fine-tuning on the collected experience recovers only 3.8\%. Compared with classical reinforcement-learning baselines, in-context learning from trial-and-error experience followed by Experience Distillation matches their performance with at least \(9.6\times\) fewer environment samples.

Problem

• sample-efficient learning
• in-context learning
• context distillation
• experience distillation
• agent learning

Innovation

• Experience Distillation
• In-Context Learning
• Sample Efficiency
• Context Distillation
• Agent Learning