Retrieval-Augmented LLM Agents: Learning to Learn from Experience

2026-03-18

AI Summary

This work addresses the limited generalization of large language model (LLM) agents on unseen tasks, a challenge exacerbated by the trade-off between adaptability and scalability in existing approaches that rely solely on fine-tuning or training-free experience retrieval. To overcome this limitation, we propose the first scalable, retrieval-augmented agent training framework that systematically integrates supervised fine-tuning with experience retrieval. Our method leverages parameter-efficient LoRA fine-tuning while incorporating optimized strategies for trajectory storage, querying, and selection to inject relevant historical experiences into the agent’s context during training, thereby enabling genuine “learning from experience.” Experimental results demonstrate that our approach significantly outperforms current baselines and achieves superior generalization on previously unseen tasks.

Technology Category

Search and Optimization: Learning to SearchMachine Learning: Large Multimodal Models (LMMs)Multiagent Systems: Multiagent Learning

Application Category

Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved informationSemantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactionsUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

While large language models (LLMs) have advanced the development of general-purpose agents, achieving robust generalization to unseen tasks remains a significant challenge. Current approaches typically rely on either fine-tuning or training-free memory-augmented generation using retrieved experience; yet both have limitations: fine-tuning often fails to extrapolate to new tasks, while experience retrieval often underperforms compared to supervised baselines. In this work, we propose to combine these approaches and systematically study how to train retrieval-augmented LLM agents to effectively leverage retrieved trajectories in-context. First, we establish a robust supervised fine-tuning (SFT) recipe using LoRA that outperforms several state-of-the-art agent training pipelines. Second, we provide a detailed analysis of key design choices for experience retrieval, identifying optimal strategies for storage, querying, and trajectory selection. Finally, we propose a pipeline that integrates experience retrieval into the fine-tuning process. Our results demonstrate that this combined approach significantly improves generalization to unseen tasks, providing a scalable and effective framework for building agents that learn to learn from experience.

Problem

• retrieval-augmented
• large language models
• generalization
• experience learning
• LLM agents

Innovation

• Retrieval-Augmented LLM
• Supervised Fine-Tuning
• Experience Retrieval
• LoRA
• In-Context Learning