Otap:Structure-Aware Optimal Transport for Evaluating Planning and Execution in Agent Trajectories

2026-07-19

AI Summary

Existing methods for evaluating agent trajectories rely on binary success flags or exact matches to reference trajectories, making them unable to distinguish between reasonable solutions and lucky successes and highly sensitive to step reordering and granularity differences. This work proposes OTAP, a pseudo-metric that formulates trajectory evaluation as a structure-aware distance between an execution graph and a set of valid solution graphs. Built upon attribute dependency graphs, OTAP leverages unbalanced fused Gromov–Wasserstein optimal transport combined with soft-coupling matching. It exhibits dependency-preserving permutation invariance, bounded sensitivity to redundant steps, and naturally accommodates missing actions, hallucinations, and granularity mismatches. Experiments demonstrate that OTAP significantly differentiates valid from invalid trajectories under controlled perturbations and across three benchmarks, substantially outperforming semantic metrics and achieving optimal performance when the underlying dependency graph is accurately recovered.

Abstract

Large language model agents solve tasks by generating trajectories that interleave planning, tool calls, and intermediate results. Current evaluation metrics reduce such a trajectory to a binary success flag or compare it against a reference by exact matching. A success flag cannot distinguish a sound solution from one that succeeds by luck, and says nothing about why a failed run went wrong. Exact matching penalizes plans that are valid but reordered or decomposed differently from the reference. We reframe trajectory evaluation as a distance between the agent's execution graph and a set of valid solution graphs, and instantiate it via an unbalanced fused Gromov-Wasserstein transport problem over attributed dependency graphs. The resulting score, termed \otap{} (Optimal Transport for Agentic Planning), is a pseudo-metric that is provably invariant to dependency-preserving reorderings and has bounded sensitivity to redundant steps. Its unbalanced marginals handle missing or hallucinated steps without forcing a match, and its soft coupling accommodates variation in plan granularity. On controlled perturbations and three public benchmarks, \otap{} separates valid from invalid trajectories in a regime where semantics-only metrics score below chance. Its accuracy is highest when the dependency graph is recovered exactly, and drops only when the graph is inferred heuristically from free-text traces.

Problem

• trajectory evaluation
• large language model agents
• planning
• execution graph
• evaluation metrics

Innovation

• Optimal Transport
• Structure-Aware Evaluation
• Agent Trajectories
• Gromov-Wasserstein Distance
• Dependency Graphs