ExpGraph: Model-Agnostic Experience Learning with Graph-Structured Memory for LLM Agents

2026-05-28

AI Summary

This work addresses the limitations of current large language model (LLM) agents in effectively reusing historical experiences and their inflexibility when switching underlying models. To overcome these challenges, the authors propose a model-agnostic experiential learning framework that organizes skills and lessons into a graph-structured external memory capable of self-evolution. The framework enables efficient retrieval through a combination of graph diffusion mechanisms and utility-aware ranking, augmented by a lightweight reinforcement learning–trained retrieval collaborator. Notably, it facilitates cross-task and cross-model experience reuse without requiring any updates to the LLM parameters. Experimental results demonstrate performance improvements of up to 12.2% in static tasks and 21.4% in agent-based environments, along with a reduction of up to 21.6% in interaction steps.

Abstract

Large language model (LLM) agents have shown strong capabilities in reasoning, tool use, and multi-step interaction, but they often solve tasks from scratch and fail to reuse successful strategies or failure lessons from prior experience. Fine-tuning on collected experience can improve reuse, but it is inflexible when stronger or more suitable executors emerge. We propose ExpGraph, a model-agnostic experience learning framework that enables frozen and replaceable LLM executors to improve through external experience reuse without parameter updates. ExpGraph summarizes historical trajectories into reusable skills and failure lessons, organizes them as nodes in a self-evolving experience graph, and retrieves useful experiences through graph diffusion and utility-aware ranking. A lightweight retrieval copilot is trained with reinforcement learning using feedback that compares executor performance with and without retrieved experiences, while the graph is updated online from downstream task outcomes. We evaluate ExpGraph on ExpSuite, covering question answering, mathematical reasoning, code generation, and multi-step agentic environments including ALFWorld and AppWorld. ExpGraph improves over the strongest baseline by 12.2% and 4.7% on static tasks with smaller and larger executors, and by 21.4% and 12.7% in agentic environments, while reducing average interaction steps by 12.7% and 21.6%. Ablations show that graph-structured experience, utility-aware ranking, and adaptive retrieval jointly enable effective experience reuse across diverse tasks and executor models.

Problem

• experience reuse
• LLM agents
• memory
• trajectory
• skill generalization

Innovation

• model-agnostic
• graph-structured memory
• experience reuse
• utility-aware ranking
• reinforcement learning