PATH-Bench: Path-Dependent Evaluation of Lifelong Agents

2026-08-02

AI Summary

Existing lifelong learning benchmarks overlook how the sequential accumulation of experience influences knowledge transfer and forgetting. This work proposes PATH-Bench, the first benchmark to systematically incorporate a path-dependence perspective: it estimates directed inter-task relationships via multi-model in-context learning, constructs probe task sequences with controllable beneficial and interfering histories, and introduces a Selective Experience Use (SEU) mechanism that dynamically filters relevant past experiences. Experiments reveal that the utility of historical experience is jointly governed by representation schemes and task interaction structures. SEU significantly enhances forward transfer while mitigating catastrophic forgetting, demonstrating consistent effectiveness across diverse tasks such as code generation and tool usage.

Abstract

Lifelong LLM agents increasingly adapt through external learning states that store past interactions as retrievable memories or reusable skills, yet existing benchmarks rarely account for how the path of accumulated experience shapes what agents transfer and retain. In this work, we establish PATH-Bench, a benchmark for path-dependent evaluation of lifelong agents. PATH-Bench estimates directed task relationships via multi-model in-context learning, constructs probe-centered sequences with controlled helpful and interfering histories, and repeatedly evaluates probe tasks to measure average performance, forward transfer, backward transfer, and forgetting. We evaluate eight representative agents on single-turn code generation and multi-turn tool-use tasks under positive- and negative-dominant histories. Benchmark results show that experience utility depends jointly on how experience is represented and on the task's interaction structure, that strong transfer does not ensure retention, and that later experience can reshape gains acquired earlier in the learning path. Based on these findings, we propose Selective Experience Use (SEU), an agent harness that regulates how path-accumulated experience influences each new task, admitting helpful items while filtering out potential interference. SEU consistently reduces forgetting while improving forward transfer in the majority of settings. The PATH-Bench provides both a controlled evaluation framework and actionable guidance for designing more selective and robust lifelong agents.

Problem

• lifelong agents
• path-dependent evaluation
• experience transfer
• forgetting
• task sequences

Innovation

• path-dependent evaluation
• lifelong agents
• forward/backward transfer
• Selective Experience Use
• experience interference