APTInvestBench: Evaluating Autonomous APT Investigation under Varying Telemetry

2026-09-30

AI Summary

This study addresses the insufficient robustness of LLM-based agents in Advanced Persistent Threat (APT) investigations under heterogeneous telemetry environments. We propose a cross-telemetry robustness benchmark and an action-level support requirement metric, combined with large-scale log simulation to construct a reusable autonomous investigation evaluation framework. This approach effectively distinguishes telemetry limitations from forensic gaps, revealing citation support instability obscured by aggregate coverage metrics. Experimental results demonstrate that agents achieve an average evidence acquisition rate of 44.3% and a formal citation support rate of only 25.0%, with 35.5% of covered actions experiencing support loss. These findings provide critical insights for enhancing the reliability of security investigation agents.

Abstract

Large language model (LLM) agents could help security operations centers (SOCs) investigate advanced persistent threats (APTs) by turning weak leads into evidence for intrusion scoping and response. Yet success under one telemetry setting does not establish robustness to changes in log collection, retention, or sampling. We introduce APTInvestBench, a benchmark for evaluating cross-telemetry robustness in autonomous APT investigation. It comprises 370 cases across seven SOC-inspired conditions, derived from 56 report-informed attack reconstructions with 16.4 million log records. Agents investigate unverified leads and submit reports with record-level citations. Fixed action-level support requirements track sufficient evidence across available logs, query returns, and formal citations, separating telemetry limitations from acquisition and reporting gaps. Across eleven LLMs, agents acquire sufficient evidence for 44.3% of recoverable attack actions on average, while formal citations support only 25.0%. More importantly, aggregate coverage can conceal substantial instability: from Full to endpoint-only telemetry, coverage declines by only 1.6 percentage points, yet 35.5% of previously covered actions lose sufficient citation support despite remaining recoverable. Across four frameworks, such losses persist even when registered supporting records remain unchanged. APTInvestBench provides reusable investigation environments and diagnostic evaluation for identifying these gaps and developing more reliable defensive agents.

Problem

• Advanced Persistent Threats
• Autonomous Investigation
• Telemetry Robustness
• Large Language Models
• Benchmark Evaluation

Innovation

• APTInvestBench
• Cross-telemetry Robustness
• Autonomous APT Investigation
• LLM Agents
• Diagnostic Evaluation