runtime tracing

Existing debugging methods for code agents struggle to trace state transitions and error propagation in complex tasks and lack scalability. This work proposes a traceable architecture that employs an evolutionary extractor to parse heterogeneous runtime logs, constructs a hierarchical trajectory tree integrated with persistent memory, and introduces a fault-origin localization algorithm. For the first time, this approach enables automatic reconstruction of the complete state transition history and precise error tracing for agents executing multi-stage, parallel tool invocations. Evaluated on the CodeTraceBench benchmark, the method significantly outperforms existing baselines and supports accurate replay of original failing execution paths through diagnostic signal regeneration.

Large language model (LLM) agents face security risks from prompt injection attacks during tool invocation. Method: This paper proposes a program-analysis-based defense framework that models the agent’s runtime execution trace as a structured graph intermediate representation—unifying control-flow, data-flow, and program-dependence graphs—and introduces a lightweight type system for static verification of sensitive data flows and trust boundaries. A graph constructor and security metadata registration mechanism enable fine-grained policy enforcement. Contribution/Results: Evaluated on the AgentDojo benchmark, our approach achieves 95.75% true positive rate and only 3.66% false positive rate, significantly outperforming existing defenses. This work pioneers the systematic application of classical program analysis techniques to formal security verification of LLM agents, establishing a new methodological pathway for agent-level safety assurance.

This work addresses the challenge of fault diagnosis and root cause analysis in complex multi-agent systems, where execution traces are often lengthy and structurally intricate, leading existing methods to either overlook critical behavioral details or focus solely on final outcomes. To overcome these limitations, the authors propose TraceSIR, a novel multi-agent framework that introduces TraceFormat—a structured abstraction for system trajectories—and integrates three specialized agents: StructureAgent for trajectory compression, InsightAgent for fine-grained root cause diagnosis, and ReportAgent for generating cross-task actionable reports. Experimental evaluation on the newly curated TraceBench benchmark demonstrates that TraceSIR significantly outperforms existing approaches in report coherence, informativeness, and actionability, thereby fulfilling the diagnostic requirements of real-world scenarios.

Existing intelligent agents often suffer from high system fragility and substantial execution overhead in multi-tool coordination due to inadequate scheduling mechanisms. This work proposes a hierarchical orchestration paradigm that obviates the need for fine-grained dependency graphs by providing coarse-grained global guidance, coupled with context-constrained intra-layer execution. A pattern-aware local reflection mechanism is introduced to enable runtime error detection and repair without triggering costly global replanning. The approach significantly enhances the robustness of tool invocation while reducing execution complexity and resource consumption, yielding a lightweight and reusable tool orchestration component.

Current LLM agents lack reproducible, intervenable, and verifiable debugging mechanisms when encountering tool failures such as timeouts, stale data, or description contamination. This work proposes the first fault reproduction–intervention–verification workflow tailored for the Model Context Protocol (MCP), implemented in an open-source web-based workbench. The platform supports real tool-call recording, injection of 12 failure types, cache-matched replay, and real-time retry capabilities. By integrating deterministic rules with an LLM-based adjudicator, the framework enables controllable behavior reproduction and rigorous performance validation. In experiments across five agents and 120 scenarios, the strongest agent completed 105 tasks; notably, the retry mechanism boosted success rates from 30% to 100% for timeout errors, while handling stale data remains challenging.

Existing static analysis tools struggle to capture novel dependencies introduced by frameworks in large language model (LLM) agent programs—such as those stemming from prompts, tools, memory, and multi-agent orchestration. This work proposes AgentFlow, the first static analysis framework tailored for agent programs, which formalizes a framework-agnostic Agent Dependency Graph (ADG) to uniformly model inter-component dependencies, control flow, and data flow by abstracting semantic elements into typed nodes and edges. Leveraging static analysis, AST parsing, and framework-specific semantic modeling, AgentFlow implements an extensible ADG construction engine adaptable to mainstream agent frameworks. Evaluated on AgentZoo—a dataset of 5,399 real-world agent programs—AgentFlow substantially improves entity and dependency identification, generates high-fidelity software bills of materials, and uncovers 238 instances of taint propagation risks from prompts to tools.

This work addresses the challenge that temporal logs of large language model agents executing complex tasks often obscure data dependencies between actions and intermediate artifacts, hindering comprehension, debugging, and reuse. To overcome this limitation, the authors propose a computation provenance–based structured representation that transforms raw execution trajectories into provenance graphs, where tool invocations are modeled as computational actions and their inputs and outputs as data artifacts. By leveraging graph alignment, quotient graph construction, and visualization techniques, the approach enables, for the first time, cross-trajectory data flow alignment, explicit dependency revelation, and automatic extraction of reusable skill patterns. Experimental results demonstrate that the method effectively uncovers hidden dependencies, aligns heterogeneous execution paths, and abstracts generalizable tool usage patterns.

This work addresses critical security vulnerabilities in multi-agent systems arising from prompt injection attacks and failures at instruction/data boundaries, which can lead to data leakage and tool misuse—particularly challenging to mitigate in heterogeneous agent workflows spanning diverse codebases. The paper introduces the first automated pre-deployment defense framework tailored for multi-agent applications. By statically analyzing prompt templates, tool interfaces, and invocation code, the framework identifies high-risk leakage patterns and synthesizes minimally invasive patches, including boundary sanitization, allowlist-based gating, and least-privilege checks. Validated against both adversarial and benign inputs, the approach ensures functional integrity without runtime overhead. Empirical evaluation on five real-world applications and the AgentDojo benchmark demonstrates complete prevention of data leakage under basic attacks and a 91% reduction under stress-induced attacks, all while preserving original system functionality.

This work addresses the challenge of locating cross-procedural vulnerability-triggering statements—specifically, identifying code that transforms program state into unsafe operations, particularly when situated deep within call chains outside patched functions. To this end, it proposes the first static analysis method integrating agent-guided exploration with causal evidence verification. The approach leverages large language model (LLM) agents to navigate code property graphs, while a deterministic acceptability gate ensures that all reported triggers are supported by explicit causal evidence. The study introduces SinkTrace-Bench, the first benchmark dataset of source-to-sink causal chains, comprising 1,542 balanced samples. Evaluated on InterPVD, the method achieves 75.0% VulnHit and 80.8% FuncHit, substantially outperforming existing techniques, and further exposes inherent limitations of LLMs in causal reasoning.

Current large language model (LLM) agents struggle to precisely localize harness defects responsible for unreliable behaviors within failed execution trajectories, leading to broad and inefficient remediation strategies. This work proposes HarnessFix, a novel framework that enables the first precise diagnosis and structured repair of harness defects based on execution traces. By constructing a harness-aware trajectory intermediate representation (HTIR), HarnessFix integrates step-level provenance tracking, control-flow analysis, and defect aggregation to fine-grainedly attribute faulty behaviors to specific steps and harness components, subsequently generating specification-guided repair patches. Experimental results demonstrate that HarnessFix achieves performance gains of 15.2%–50.0% across four benchmarks, including SWE-Bench Verified, significantly outperforming both handcrafted and self-evolution baselines, while uncovering recurrent harness defect patterns in the ETCLOVG architecture.