DiagChain: A Diagnostic Benchmark for Evaluating LLM Agents on Evidence-Grounded Attack Chain Reconstruction

2026-08-04

AI Summary

This work addresses a critical limitation in existing attack chain reconstruction benchmarks, which focus solely on final outcomes and fail to pinpoint intermediate reasoning errors. To enable fine-grained evaluation, we introduce MAIN-69—the first staged diagnostic benchmark comprising 69 scenarios and 849 reference steps—combined with Evidence-Centric Retrieval-Augmented Generation (ECRAG) and a structured attack chain representation. We evaluate LLM agents across diverse operating systems, noise levels, and chain lengths using five complementary metrics, revealing that even the strongest model achieves only a 39.6% success rate: smaller models struggle to integrate evidence effectively, while larger models are hindered by suboptimal evidence ordering. Our approach establishes an interpretable and diagnosable paradigm for attack chain reasoning.

Abstract

Large Language Model (LLM) agents offer a promising approach to attack chain reconstruction by retrieving and interpreting heterogeneous telemetry to infer ordered attacker actions. However, existing benchmarks mainly evaluate final outputs or aggregate accuracy, providing limited insight into how errors arise and propagate across intermediate reasoning stages. We present DiagChain, a diagnostic benchmark for evidence-grounded attack chain reconstruction that enables stage-wise evaluation of LLM agents. DiagChain includes MAIN-69, a suite of 69 scenarios spanning multiple operating systems, evidence noise levels, and chain lengths. It further introduces Evidence-Centric Retrieval-Augmented Generation (ECRAG), which couples evidence retrieval with an evolving structured representation of the reconstructed chain. Five complementary metrics are introduced to assess distinct stages of the reconstruction process and support systematic failure diagnosis. Based on evaluations using 6 LLMs, DiagChain reveals that even the strongest configuration succeeds on only 39.6% of the 849 reference steps in MAIN-69. Our analysis further shows that smaller models struggle with the more basic task of incorporating retrieved evidence into their outputs, whereas larger models can proceed to later steps, where correctly ordering that evidence becomes the main bottleneck. These results validate the importance of diagnostic evaluation beyond end-to-end accuracy and provide actionable insights for improving evidence-grounded cybersecurity agents.

Problem

• attack chain reconstruction
• diagnostic benchmark
• evidence-grounded reasoning
• LLM agents
• intermediate reasoning errors

Innovation

• attack chain reconstruction
• diagnostic benchmark
• evidence-grounded reasoning
• retrieval-augmented generation
• LLM agent evaluation