Beyond Token-Level Cross-Entropy: Fréchet Distributional Post-Training for Autoregressive Image Generation

2026-08-01

AI Summary

This work addresses the mismatch between the token-level cross-entropy training objective and image-level distribution quality evaluation in autoregressive image generation. The authors propose FD-loss, a post-training method that, for the first time, directly optimizes discrete autoregressive models using the image-level Fréchet distance. By introducing a dual-channel mechanism to construct gradient-free rollout contexts and employing a probability-level straight-through estimator for differentiable replay, the approach updates only the generator without adding parameters or inference overhead. Evaluated on ImageNet at 256×256 resolution, the method reduces the average FID and FD₆ by 41.4% and 52.0%, respectively, and improves the best FID from 2.42 to 1.43, effectively bridging the gap between training objectives and evaluation metrics.

Abstract

Autoregressive image generators are commonly pretrained with token-level cross-entropy under teacher forcing, yet evaluated by the distributional quality of decoded images. This creates an objective mismatch, because categorical errors have unequal image-level consequences, and a context mismatch, because inference conditions on model-generated histories. We introduce FD-loss post-training, which adapts a pretrained discrete generator using representation-space Fréchet distance as the sole objective. A dual-pass scheme first constructs detached rollout contexts through gradient-free generation under the model's native inference configuration, then performs differentiable replay with a probability-level straight-through estimator (STE) that preserves hard argmax decoding in the forward pass while propagating image-level gradients through temperature-scaled probabilities. Only the generator is updated, while the tokenizer and feature extractors remain frozen. Across eight completed configurations from four generator families on class-conditional ImageNet at $256\times256$, FD-loss post-training reduces FID and $\mathrm{FD}_{r6}$ by 41.4% and 52.0% on average. The strongest FID result improves from 2.42 to 1.43 without adding parameters or inference steps.

Problem

• objective mismatch
• context mismatch
• autoregressive image generation
• distributional quality
• token-level cross-entropy

Innovation

• Fréchet distance
• post-training
• autoregressive image generation
• straight-through estimator
• objective alignment