Divergence Decoding: Training-Free Capability Fusion

2026-07-28

AI Summary

This work addresses the trade-off between domain expertise and robust reasoning in large language models: general-purpose models lack specialized knowledge, while domain-specific models often exhibit weak reasoning capabilities and poor robustness. To reconcile these limitations without additional training, the authors propose a novel inference-time capability fusion framework that reinterprets speculative decoding’s “draft–verify” mechanism as an adaptive routing strategy based on distributional divergence. Specifically, the method dynamically allocates control between a general and a domain-specific model by monitoring their output distribution discrepancies in real time using Jensen–Shannon divergence, enabling seamless collaboration. Evaluated on scientific benchmarks such as GPQA and ChemBench, the approach significantly outperforms individual models, demonstrating consistent performance gains on complex scientific reasoning tasks.

Abstract

While large language models excel in reasoning, these generalists often lack knowledge for specialized scientific domains. Conversely, domain models~(specialists), while knowledgeable, suffer from specialization side-effects including diminished logic and reduced robustness.To address this dilemma, we introduce Divergence Decoding, a training-free framework for capability fusion. It reconstructs the "draft-and-verify" skeleton of speculative decoding into an adaptive routing mechanism. The core is using Jensen-Shannon divergence to monitor the distributional disagreement between the two models at each token. When the specialist exhibits significant divergence, our method identifies it as a potential reasoning risk and instantaneously routes control to the generalist. This allows the dynamic injection of general reasoning while preserving domain expertise, achieving inference-time policy composition of the generalist and the specialist.We evaluate Divergence Decoding across diverse model families (Qwen and Llama series) on challenging scientific benchmarks (GPQA, ChemBench, and ChemCoTBench). Experimental results demonstrate that Divergence Decoding outperforms both the domain-specialized and general-purpose models, effectively surpassing the performance of most single-model baseline. This suggests that Divergence Decoding provides a general, training-free paradigm for fusing diverse LLM capabilities through adaptive inference-time collaboration.

Problem

• large language models
• domain specialization
• reasoning capability
• model fusion
• inference-time collaboration

Innovation

• Divergence Decoding
• training-free
• capability fusion
• adaptive routing
• Jensen-Shannon divergence