AOSpec: Action and Observation Co-Speculation for Low-Latency Agent Serving

2026-08-01

AI Summary

This work addresses the latency bottleneck in large language model (LLM) agents caused by the serial generation–execution pipeline when interacting with stateful tools. To overcome this, the authors propose AOSpec, a novel framework that introduces joint action–observation speculation. By leveraging value-aware decoding to guide observation prediction and combining isolated execution branches with joint action–state verification, AOSpec achieves low-latency, lossless service while preserving semantic consistency. The approach reframes long-range dependencies as targeted action–state validation, effectively breaking the trade-off between speculation horizon and accuracy. Experiments on Terminal-Bench demonstrate 11.8–32.5% reduction in end-to-end average latency and up to 42.8% lower p99 latency, with the observation model exhibiting cross-task transferability without retraining.

Abstract

Large language model agents increasingly act through stateful tools, yet model generation and environment execution remain serialized at every step. As decoding accelerates, tool execution becomes a growing bottleneck. Existing action- or observation-only speculation leaves much of this latency exposed: value is concentrated in a few slow calls, some outcomes emerge only through execution, and longer lookahead typically requires an increasingly unlikely chain of action predictions. We present AOSpec, a lossless framework that co-speculates actions and observations across the full agent-environment loop. Expected Value Decoding (EVD) directs observation speculation toward outcomes with the greatest expected latency benefit, optimizing expected time hidden rather than hit rate. For outcomes only execution can reveal, AOSpec launches latency-critical target actions in isolated forks that contain their effects, while Joint Action-State Verification (JASV) verifies both the action and its origin state against committed execution before reuse. JASV recasts long-horizon action dependency from full-chain prediction into target action-state verification, breaking the lookahead--accuracy tradeoff and unlocking long-range overlap without sacrificing serial semantics. Across Terminal-Bench serving settings spanning four harnesses, five actor models, and five serving speeds, AOSpec outperforms every practical baseline, reducing mean end-to-end latency by 11.8-32.5% and p99 latency by up to 42.8%. Its gains increase as decoding accelerates, and its observation model transfers from Terminal-Bench to SWE-bench Verified without retraining.

Problem

• low-latency agent serving
• action-observation speculation
• tool execution bottleneck
• serial semantics
• end-to-end latency

Innovation

• co-speculation
• Expected Value Decoding
• Joint Action-State Verification
• low-latency agent serving
• action-observation speculation