OoO-Spec: Out-of-Order Semantic Speculation for Fast Tool Calling

2026-08-01

AI Summary

This work addresses the high latency and lack of parallelism in existing large language models that generate tool calls token-by-token, preventing concurrent prediction of functions and their parameters. To overcome this, the authors propose a lightweight sidecar model that, upon request arrival, simultaneously predicts both the function choice and all parameter slots in parallel. The predictions are then seamlessly integrated into the main model’s decoding process via non-blocking semantic injection. This approach achieves, for the first time, request-level out-of-order semantic speculation without requiring retraining of the drafter for each target model, offering strong generality and multi-model deployment capability. Evaluated with a LoRA-finetuned Qwen3-0.6B sidecar model combined with split-GPU pipeline parallelism, the method yields an average 3.89× speedup across 21 target-benchmark combinations—significantly outperforming ToolSpec (2.95×) and existing learned drafters by 34.1%.

Abstract

LLMs generate tool calls token by token, even though the function choice and argument values can often be predicted in parallel from the request and tool schema. ToolSpec reduces this cost by drafting schema tokens and retrieving earlier calls, but cannot propose request-specific values absent from either source. We present OoO-Spec, which computes these missing semantics out of order. At request arrival, a Qwen3-0.6B sidecar predicts the function choice and all schema-defined argument slots in one parallel request-level wave while the target begins ToolSpec decoding. The runtime joins the slot values, renders the resulting call as text, and exposes it to subsequent candidate-construction rounds. The target polls without blocking, re-tokenizes a ready hint with its own tokenizer, and remains the sole verifier and commit authority. The sidecar is trained once with LoRA on Qwen2.5-32B teacher traces and used unchanged across Qwen2.5, Qwen3, and Llama targets, without target-specific drafter training. Across seven fully ranked targets and three benchmarks under greedy batch-one decoding, OoO-Spec is fastest among all evaluated methods in all 21 target-benchmark cells, reaching 2.46x-5.34x over autoregressive decoding with an unweighted mean of 3.89x, versus 2.95x for ToolSpec. It also outperforms every evaluated released learned drafter in each comparable cell. Across Qwen3-4B, 8B, 14B, and 32B targets, the same sidecar improves on ToolSpec by 34.1% on average. Its compact semantic payload averages 85 bytes per request excluding protocol metadata, supporting effective split-GPU overlap.

Problem

• tool calling
• semantic speculation
• out-of-order execution
• large language models
• parallel decoding

Innovation

• out-of-order speculation
• tool calling acceleration
• sidecar model
• schema-aware parallel prediction
• LoRA-based transfer