Knowledge Distillation for Visual Autoregressive Models

2026-06-04

AI Summary

This work addresses the limited efficacy of existing knowledge distillation methods for visual autoregressive (AR) image generation models, which stems from long-sequence decoding and ambiguity in visual tokens. The study presents the first systematic investigation into distillation mechanisms for visual AR models and introduces VarKD, a novel framework that selects high-quality samples via a student-aware sampling strategy and applies selective teacher supervision at the token level to mitigate unreliable supervisory signals. Tailored to the characteristics of image generation, VarKD consistently outperforms current distillation approaches across diverse AR backbone architectures on ImageNet, substantially narrowing the performance gap between compact student models and large-scale teacher models.

Abstract

Autoregressive (AR) image generation models are highly expressive but computationally intensive, motivating effective model compression. Knowledge distillation (KD) is a natural approach for model compression and has been widely studied in language modeling, yet its behavior in visual AR generation remains underexplored. In this work, we present the first systematic study of distillation strategies for AR image models. Our analysis shows that while standard distillation can yield meaningful gains, recent methods developed for language do not directly transfer to images: long decoding horizons and visual token ambiguity make teacher supervision unreliable especially under student-conditioned contexts. To address this, we propose VarKD, a distillation framework for visual autoregressive models that distills on student samples while selectively applying teacher supervision and reducing token-level ambiguity. Experiments on ImageNet across multiple AR backbones show that VarKD consistently outperforms prior distillation baselines, narrowing the gap to large-scale models.

Problem

• Knowledge Distillation
• Visual Autoregressive Models
• Model Compression
• Image Generation
• Token Ambiguity

Innovation

• Knowledge Distillation
• Visual Autoregressive Models
• VarKD
• Token Ambiguity
• Model Compression