AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs

2026-09-28

AI Summary

This study addresses the limitation of existing LLM inference benchmarks in reflecting real-world agent workloads involving multi-turn interactions, growing contexts, and tool invocations. We present the first systematic quantification of the performance gap between chat and agent scenarios by constructing an agent inference benchmark suite. Specifically, we generate synthetic workloads by sampling from real execution traces such as SWE-Bench, and employ Nsight Compute kernel-level profiling combined with a multidimensional Roofline model to precisely identify hardware bottlenecks. Our analysis accurately characterizes GPU resource utilization and reveals potential bottlenecks of emerging hardware architectures under diverse agent workloads. These findings provide a reliable empirical foundation for optimizing inference engines tailored to agentic applications.

Abstract

The optimization of LLM serving engines, such as vLLM and SGLang, is largely benchmark-driven: optimizations, scheduling policies, hardware and system designs are all selected based on representative workloads. However, a significant mismatch has emerged in the agentic era. Existing benchmarks primarily focus on simple single-turn chatbot workloads. LLM applications are increasingly agentic: coding agents, terminal execution systems, and tool-use agents issue multi-turn requests with growing context lengths. We introduce AgentPerfBench, a benchmark suite for agentic inference. It uses real traces from agentic benchmarks, such as SWE-Bench and TerminalBench, alongside standard chat baselines. This enables benchmarking of models on multi-turn tasks involving tool calling, skill utilization, and increasing context lengths. AgentPerfBench also samples from empirical distributions of input length, output length, and turn count derived from the real traces, generating representative synthetic profiles for cheap and accurate measurements on new hardware. In addition, we further find that several existing benchmarks fail to accurately reflect real hardware performance for two key reasons: 1) they do not account for realistic context-length growth, and 2) they measure inference performance without operating at hardware saturation. We discuss these issues in detail and provide rich kernel-level Nsight Compute (NCU) traces to construct a new multi-dimensional roofline model that captures hardware-system limitations in both memory bandwidth and memory capacity footprint. The benchmarking suite then includes automated scripts to identify potential bottleneck conditions on emerging hardware when evaluated with diverse agentic traces. Together, these contributions quantify the chat-to-agentic gap in current inference benchmarks and characterise per-kernel GPU resource utilisation via roofline analysis.

Problem

• LLM inference benchmarking
• agentic workloads
• multi-turn requests
• context length growth
• hardware saturation

Innovation

• Agentic LLM Inference
• Benchmark Suite
• Roofline Model
• Synthetic Workload Generation
• Multi-turn Context