FutureBridge: Token Selection Beyond Local Preference in Collaborative Decoding

2026-08-07

AI Summary

This work addresses a key limitation in existing collaborative decoding methods, which rely solely on large language models’ (LLMs’) local token preferences while neglecting the coherence of subsequent reasoning by small language models (SLMs). To overcome this, the authors propose a token selection mechanism that anticipates the SLM’s future reasoning capabilities. Specifically, they construct LLM reasoning trajectories grounded in answer verification to generate a shared future context, then perform counterfactual evaluation under a frozen SLM to train a lightweight reranker that depends only on the current state. During inference, the LLM expands the candidate token pool while the SLM drives generation. This approach transcends the constraints of LLM-local preferences and improves the Math Avg score of Qwen3-1.7B by 35.1% over greedy decoding across five mathematical reasoning benchmarks.

Abstract

Token-level collaboration allows a large language model (LLM) to assist a small language model (SLM) when their predictions diverge. Existing methods either use LLM-generated intervention tokens or rank candidates with the LLM's next-token probabilities. Both rely on the LLM's local preference, even though an LLM-selected token may be difficult for the SLM to build on. We present FutureBridge, which ranks joint LLM-SLM token candidates according to how well they support the SLM's subsequent reasoning. During training, an answer-verified LLM trajectory supplies a fixed shared future, and a frozen SLM evaluates every candidate under this common context. The resulting counterfactual scores supervise a lightweight token reranker that observes only the current state and candidate token. At inference, FutureBridge uses the LLM only to expand the candidate pool, selects one token, and returns generation to the SLM without generating or appending a future suffix. Across five mathematical reasoning benchmarks, FutureBridge improves the Qwen3-1.7B SLM's Math Avg. by 35.1% relative to greedy SLM decoding. These results indicate that token selection benefits from modeling whether the receiving SLM can use each candidate to continue reasoning, rather than relying on the LLM's local preference alone.

Problem

• collaborative decoding
• token selection
• small language model
• large language model
• reasoning continuity

Innovation

• collaborative decoding
• token selection
• small language model
• future-aware ranking
• counterfactual evaluation