Speculate While You Reason: Teaching Agents to Predict Their Next Tool Call via Joint Agent-Speculator RL

2026-07-28

AI Summary

This work addresses the significant latency incurred by large language model agents during tool invocation, a challenge exacerbated by existing speculative methods whose performance is limited due to behavioral misalignment with the main agent. To overcome this, we propose a self-speculative agent architecture that unifies the agent and the speculator within a single model, leveraging the agent’s own execution trajectory to generate speculation targets—thereby eliminating behavioral alignment issues. Our approach integrates joint reinforcement learning, KV cache reuse, partial-trajectory-based tool prediction, and an alternating policy update mechanism to co-optimize task execution and tool call prediction. Evaluated on Qwen3-4B and Qwen3.5-4B, our method improves Hit@1 for next-tool prediction from 44.1 to 61.2 and from 48.9 to 66.3, respectively, while maintaining task success rates.

Abstract

Large language model agents often spend substantial wall-clock time waiting for tool call results. Tool-call speculation can hide this latency by predicting and pre-executing an agent's next tool call if the prediction matches the agent's eventual tool call, but existing speculators are typically separate draft models or cached traces that are poorly aligned with the deployed agent's own behavior. We identify this speculator-agent gap and show that the target agent itself is a strong next-call speculator. This points to a simpler design: unifying the agent and speculator within the same model. In this paper, we introduce the self-speculating agent, a single model that both solves tasks in agent mode and predicts its next tool call from partial trajectories in speculator mode, fully reusing prefix KV cache. To enable this dual-mode agent without degrading performance, we propose a joint agent-speculator reinforcement learning method, which derives speculation targets from the agent's own rollouts and alternates agent and speculator updates. Across agentic search QA and conversational tool-use agentic tasks, our method improves average next tool-call Hit@1 from 44.1 to 61.2 for Qwen3-4B and from 48.9 to 66.3 for Qwen3.5-4B, while preserving agent task success.

Problem

• tool-call speculation
• agent-speculator gap
• large language model agents
• latency hiding
• speculation alignment

Innovation

• self-speculating agent
• tool-call speculation
• joint reinforcement learning
• KV cache reuse
• agentic reasoning