HAWK: Rethinking Multimodal Drafting for Speculative Decoding

2026-09-30

AI Summary

This study addresses the inefficiency of speculative decoding in large vision-language models and the difficulty drafters face in leveraging multimodal information. We propose an efficient speculative decoding method that integrates compressed visual features. Specifically, target layers are selected based on representation similarity, and multi-layer hidden states are fused to replace raw token inputs with compressed visual representations. Furthermore, we model the dynamic shift of target predictions relative to draft proposals and employ trajectory-aware distillation to optimize multi-step drafting. Experimental results demonstrate that our approach increases the average acceptance length to 4.08 on SmolVLM, achieving a 2.60× speedup and significantly outperforming the EAGLE-3 baseline.

Abstract

Speculative decoding has achieved substantial lossless speedups for LLMs, but remains less effective for large vision-language models (LVLMs), where lightweight drafters struggle to use rich multimodal information. A second limitation is that standard distillation supervises the drafter only along the original training trajectory, without modeling how target predictions shift after the drafter's own proposals. As drafting moves away from this trajectory, the drafter can increasingly disagree with the target, reducing acceptance in later steps. We propose HAWK to address both limitations. HAWK uses representation similarity to select informative target layers and learns how to combine their hidden states. For visual information, it directly provides the drafter with compressed visual hidden states from the target model instead of raw visual tokens, making the visual information easier for a shallow drafter to use. HAWK also trains the drafter to capture how target predictions change after its own proposals, improving its agreement with the target during multi-step drafting. On SmolVLM-256M across ten multimodal benchmarks, HAWK raises average acceptance length from 3.32 to 4.08 and speedup from 2.19x to 2.60x over EAGLE-3 under greedy decoding, and from 2.89 to 3.41 and 1.92x to 2.19x under sampling.

Problem

• Speculative Decoding
• Large Vision-Language Models
• Multimodal Drafting
• Draft Model
• Knowledge Distillation

Innovation

• Speculative Decoding
• Multimodal Drafting
• Vision-Language Models
• Representation Similarity
• Knowledge Distillation