Architectural Implications of Agentic AI Workflows

2026-08-05

AI Summary

Traditional unified-architecture servers struggle to efficiently support fragmented, heterogeneous, and highly bursty Agentic AI workflows, resulting in low resource utilization and high tail latency. This work presents the first systematic characterization of Agentic AI workflows, identifying key architectural traits such as execution fragmentation, role heterogeneity, and dynamic resource demand fluctuations. Building on these insights, we propose Agora, a server architecture optimized specifically for Agentic AI. Agora introduces several synergistic mechanisms, including role-aware resource pooling, dynamic CPU core reclamation, GPU memory overcommitment, and state prefetching. Experimental results demonstrate that Agora significantly improves server resource utilization and throughput while maintaining stringent tail-latency guarantees, thereby validating the efficacy of the proposed architecture.

Abstract

Agentic AI is emerging in datacenters, but its architectural implications remain unexplored. We organize agentic workflows in a taxonomy and present its first architectural characterization with a production study at Microsoft Azure and a controlled study of open-source frameworks. We show that agentic execution is fragmented and heterogeneous. Requests expand into a workflow of LLM inferences, tool invocations, and orchestration decisions that repeatedly cross the CPU-GPU boundary. Our taxonomy explains how this fragmentation turns into resource demand. As orchestration and tools run on the host, the CPU sits on the critical path. Execution structure sets the load over time, which stays low with sudden spikes. Model composition sets how evenly the workflow uses the GPUs. Diversity in tasks and tools widens this range even further. These characteristics expose architectural mismatches of conventional uniform servers. Fragmented execution strands CPU and GPU capacity despite bursty demand. Different software roles make homogeneous CPU provisioning inefficient. Finally, multiplexing many agents onto shared cores degrades microarchitectural locality. Guided by our findings, we derive implications for agentic servers and examine them through Agora, our prototype for commodity servers. Agora dynamically harvests idle CPU cores for co-located throughput work, while protecting agentic tail latency against tool spikes. It oversubscribes GPU memory by placing more agents on each GPU, prefetching the next agent's state to hide swap latency. To match the machine to the heterogeneous roles, Agora pools cores by role and applies affinity-aware scheduling to restore locality. It automatically tunes mechanisms to the workload. Agora improves utilization and server throughput while preserving agent tail latency. Our insights also identify key directions for future server architectures for agentic AI.

Problem

• Agentic AI
• architectural implications
• fragmented execution
• heterogeneous workloads
• CPU-GPU boundary

Innovation

• Agentic AI
• architectural characterization
• CPU-GPU orchestration
• resource fragmentation
• affinity-aware scheduling