trajectory analysis

This work addresses the challenge of interpreting and analyzing long-horizon autonomous agent behaviors, which are typically recorded as unstructured natural language traces that hinder efficient diagnosis of inefficiencies, errors, and compliance violations. To overcome this limitation, the authors introduce ACT*ONOMY—the first grounded theory–based behavioral taxonomy for autonomous agents—featuring a three-level hierarchical structure comprising 10 actions, 46 sub-actions, and 120 leaf categories. Accompanied by an open knowledge base and an automated annotation pipeline, ACT*ONOMY provides a scalable, shared semantic vocabulary and standardized analysis protocol. This framework enables systematic cross-agent behavior comparison, facilitates failure mode identification, and reveals behavior patterns correlated with system faults, thereby substantially enhancing human oversight and control over autonomous agents.

Current agent evaluation predominantly relies on pass rates or reward scores, which often fail to uncover underlying decision-making differences or root causes of failure. This work proposes the first cross-model shared decision landscape representation that maps multi-agent interaction trajectories into a unified state graph, characterizing behavioral patterns through three event types: visitation, exposure, and repair. This enables the identification of efficient pathways and trap regions. Building upon this representation, the authors develop an interpretable trajectory diagnosis framework and introduce a lightweight runtime recovery strategy that combines state matching with policy selection to enable trap-aware intervention. Evaluated on SWE-bench, the approach improves solution rates from 40.4% to 43.5% on a specific subset and from 41.0% to 44.8% on publicly triggered instances.

Existing explainability methods struggle to address safety and accountability challenges in agent systems arising from goal misalignment, compounding decision errors, and multi-agent coordination. This work pioneers the extension of explainability from static models to dynamic agent systems by introducing a novel framework that spans the entire lifecycle—from goal specification and environmental interaction to outcome evaluation. Integrating agent architecture analysis, temporal decision tracing, and context-aware explanations, the proposed framework establishes an explainability paradigm tailored for multi-step, interactive AI systems. It clarifies the limitations of current approaches and provides a theoretical foundation and future directions for deploying safe and accountable agents.

This work addresses the pervasive issue of unreliable evaluation and poor reproducibility in agent-based research, often stemming from missing rollout records. To remedy this, the authors propose Rollout Cards—a formal reproducibility standard centered on complete rollout trajectories—as structured containers for execution logs, scoring rules, and data exclusion criteria. This approach institutionalizes rollout documentation for the first time and is integrated into the open-source platform Ergon, enabling cross-task re-evaluation. Empirical studies across four domains demonstrate that modifying only the scoring protocol can shift performance scores by up to 20.9 percentage points and even reverse model rankings, underscoring the critical role of transparent, standardized rollout reporting in ensuring result credibility.

This work addresses the coarse-grained explanation problem of Linear Temporal Logic (LTL) rules in eXplainable AI Planning (XAIP). We propose the Rule-State Assessment (RSA) framework, the first approach to *instantaneously* classify, at each timestep along a single execution trace, the status of every LTL rule as active, satisfied, inactive, or violated. Methodologically, RSA leverages formal LTL semantics and integrates symbolic trace execution with a rule-state classification algorithm, enabling fine-grained, queryable, and semantically precise behavioral attribution. Unlike conventional trace-level summaries, RSA supports real-time, dynamic, and multi-rule-coordinated tracking of behavioral evolution. Experimental evaluation demonstrates RSA’s effectiveness as a post-hoc diagnostic tool, significantly enhancing the transparency and debuggability of agent decision-making processes.

This study addresses the challenge of extracting interpretable tactical knowledge from micro-level maneuvers in real-time strategy games, which is hindered by high-dimensional, coupled state-action sequences and opaque decision-making mechanisms. To overcome this, the authors propose SAT-RTS—a novel state-action-tactic analysis pipeline that integrates BK-tree clustering, multidimensional similarity metrics, and rule-driven multi-label extraction to abstract raw behavioral sequences into discrete, human-interpretable tactical labels. The framework further incorporates a hierarchical visualization scheme for attributing tactical decisions, substantially enhancing both the interpretability and computational efficiency of tactical analysis. By enabling fitness landscape visualization, SAT-RTS effectively uncovers latent tactical drivers operating within complex game environments.

This work addresses the challenge of balancing intervention timing in imitation learning for web agents trained on expert trajectories: late interventions risk irreversible errors, while early ones induce policy dependence and local optima. To resolve this, the authors propose SRC, a branch-level imitation learning framework that employs speculative execution with a fixed horizon and a rollback mechanism. The teacher identifies the first harmful deviation and reverts to the preceding valid action prefix. High-quality, diverse trajectories are selected via a hard verifier and stored in a lightweight quality-diversity archive to supervise fine-tuning of the next action. Evaluated on WebArena-Infinity, SRC achieves superior performance over stepwise review using only 977 full trajectories and 9,183 action samples, reducing expert query costs while enhancing error recovery and preserving multiple solution paths.

This work addresses the limitation of existing tool-calling evaluation methods, which predominantly rely on post-hoc analysis and thus cannot correct errors in real time during reasoning. To overcome this, the authors propose a dual-agent architecture featuring an independent reviewer agent that evaluates tool calls before execution, shifting the paradigm from passive correction to proactive intervention. The reviewer’s decisions are guided by a Helpfulness-Harmlessness metric that quantifies the trade-off between potential benefits and risks. Coupled with an inference-time feedback mechanism and GEPA-based automatic prompt optimization, this approach enhances system performance without requiring model retraining. Empirical results demonstrate accuracy improvements of 5.5% and 7.1% on BFCL and Tau2-Bench, respectively, with the o3-mini model achieving a benefit-to-risk ratio of 3:1; further gains of 1.5–2.8% are attributable to GEPA optimization.

This work addresses the instability and performance degradation in AI agents caused by ambiguity in natural language prompts. To mitigate this issue, the authors propose an automated prompt refinement mechanism grounded in semantic trajectory analysis. By monitoring agent execution logs and extracting semantic features to detect undesirable behaviors, the system dynamically injects corrective instructions, enabling incremental, data-driven optimization of system prompts. Integrated into the open-source Agent Mentor library, this approach synergistically combines large language models, log analysis, and dynamic prompt engineering. Empirical evaluations across diverse agent configurations and benchmark tasks demonstrate significant improvements in accuracy, with particularly pronounced gains in scenarios where initial prompts exhibit semantic ambiguity.

Current benchmarks for evaluating large language model agents remain fragmented, and single metrics fail to comprehensively capture their true behavioral capabilities. This work proposes a multidimensional evaluation framework that introduces a novel dual-layer taxonomy—comprising “six control decision states” and “nine trajectory failure types”—integrated with classification-aware and blind comparison methodologies, prompt-mode ablation studies, and fine-grained, trajectory-level annotations. The framework further establishes a six-dimensional behavioral auditing mechanism spanning fifteen existing benchmarks. Experiments across eight state-of-the-art models reveal that removing explicit label menus reduces trajectory accuracy by 14–40 percentage points, and no model simultaneously excels in control accuracy, trajectory diagnosis, and tool-context utility, exposing systematic limitations in current agent behaviors.