Problem
Research questions and friction points this paper is trying to address.
LLM agents
agent serving
production scale
workload characterization
inference infrastructure
Semantics, Workflows, and Infrastructure: Understanding Agent Serving at Production Scale
🤖 AI Summary
This study addresses the absence of an end-to-end perspective linking task initiation, workflow execution, and inference infrastructure in existing agent serving research. Analyzing 11.7 million production request logs through large-scale tracing, multi-level workload characterization, and GPU cluster evaluation, this work establishes the first correlation analysis framework spanning semantic, workflow, and infrastructure layers. The investigation reveals critical patterns including request skew, rare workflow overlap, and cross-task context reuse, while identifying key deployment bottlenecks and their performance implications for agent serving systems. Furthermore, it systematically delineates open challenges within this domain, providing an empirical foundation and directional guidance for future research and system optimization.
📝 Abstract
Large language model (LLM) agents execute applications through a workflow of inference requests with tool calls and user interactions. Serving these applications at production scale requires understanding how application behavior shapes inference demand and for guiding efficient execution. Recent characterization studies provide request-level workload measurements and agent execution analysis. However, an end-to-end view connecting task initiation, workflow execution, and inference infrastructure remains unexplored. In this paper, we analyze a two-week trace of 11.7 million requests from a large-scale production platform for general-purpose agents, backed by inference infrastructure comprising over 10k GPUs. We characterize the platform at three connected levels: task-level initiation semantics, workflow-level execution patterns, and infrastructure level serving demands. Our measurements reveal workload patterns such as highly skewed request volumes across sessions, rare execution overlap among logical sibling requests, and context reuse across task boundaries. Building on these observations, we analyze deployment implications and identify open problems to guide future research on agent serving systems.
Innovation
Methods, ideas, or system contributions that make the work stand out.
LLM Agents
Production-scale Serving
Workload Characterization
Workflow Execution
Inference Infrastructure
🔎 Similar Papers
No similar papers found.
