AgentReplay: Token-Wise Trace Replay Is Essential for Fair Serving System Performance Benchmarking

2026-09-26

AI Summary

This study addresses the benchmarking unfairness in LLM agent serving evaluation caused by divergent execution trajectories. We propose a configurable record-and-replay framework that introduces, for the first time, a token-level precise replay mechanism to decouple trajectory generation from performance evaluation. By leveraging token-level state recording, Mixture-of-Experts (MoE) routing control, tool latency simulation, and autoregressive computation, our approach enforces the exact reproduction of original token sequences and expert selections, thereby supporting consistent replay of long-horizon tasks across different models. Experimental results demonstrate that this framework effectively eliminates workload variance, achieving fairer server performance comparisons than greedy decoding and length-matched replay baselines.

Abstract

LLM-based agents execute multi-turn workflows with interleaved model inference and tool calls, making efficient serving increasingly important. However, evaluating serving optimizations is challenging because identical tasks can produce different execution trajectories. Changes in generated tokens can alter subsequent prompts, tool calls, and reasoning turns, making it difficult to distinguish system improvements from workload variation. Greedy decoding does not guarantee identical outputs, while replaying only sequence lengths loses token information that affects prefix caching and mixture-of-experts (MoE) routing. To address these problems, we propose AgentReplay, a configurable trace record-and-replay framework for agent serving. AgentReplay records input/output tokens, expert selections, request dependencies, and tool durations. During replay, it forces the recorded output tokens while performing normal autoregressive computation, with optional controls for MoE expert selection and tool delays. This allows different systems to execute the same recorded workload while retaining their own batching, scheduling, and parallelization decisions. We further separate trajectory generation from performance evaluation, enabling compatible smaller models to replay long-horizon traces collected with more capable models. Our experiments show that token-wise replay in AgentReplay effectively eliminates workload variation that greedy decoding and length-wise replay cannot avoid, enabling fairer performance comparisons across serving configurations.

Problem

• LLM serving
• performance benchmarking
• agent workflows
• workload variation
• trace replay

Innovation

• Trace Replay
• Serving Benchmarking
• LLM Agents
• Mixture-of-Experts
• Token-wise Replay