Oilbird: Training-Free Speculative Decoding with Keys the Verifier Already Computes

2026-08-04

AI Summary

Existing training-free speculative decoding methods rely on exact lexical suffix matching, which struggles to retrieve semantically correct but lexically divergent candidate sequences—particularly limiting their effectiveness in tool-call scenarios. To address this, this work proposes constructing semantic keys from the verifier’s already-computed hidden states to re-index the context pool and integrate it with the conventional lexical draft tree, forming a dual-source drafting mechanism. This approach incurs no additional training or computational overhead while substantially improving draft acceptance rates. Across three mainstream drafters under identical pool sizes and budget constraints, it achieves a 24–29% increase in accepted sequence length. On the API-Bank benchmark, it attains a 4.4× speedup over autoregressive decoding, outperforming both the strongest training-free baseline (3.9×) and EAGLE-3 (2.0×).

Abstract

Training-free speculative decoding drafts by matching an exact suffix of the context against a pool of earlier context. That lookup misses correct drafts already in the pool, most visibly on tool-calling traffic, where a request repeats almost everything but the few values minted for it, and where one rejected token discards the correct continuation behind it. We diagnose the failure position by position across ten benchmarks and find it to be a problem of addressing rather than of coverage: on our densest tool-calling benchmark, about half of what the strongest exact-match drafter misses is present in the pool yet unreachable by exact matching. We therefore propose a second, semantic draft source: the same pool, re-keyed by the hidden state the verifier has already computed at each committed token, together with a merge that lets it ride inside an existing lexical drafter's tree. In three published drafters, at matched pool and budget, it lifts accepted length by 24-29%. Oilbird reaches 4.4x autoregressive decoding speed on API-Bank, against 3.9x for the strongest training-free baseline in our harness and 2.0x for EAGLE-3.

Problem

• speculative decoding
• training-free
• exact matching
• tool-calling
• draft generation

Innovation

• training-free speculative decoding
• semantic draft source
• hidden-state re-keying
• tool-calling optimization
• autoregressive acceleration