Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers

2026-07-30

AI Summary

This work addresses the intractable quadratic computational complexity of full attention mechanisms in high-resolution image and long video generation. To overcome this, the authors propose Chimera, a hybrid vision diffusion backbone that unifies text, image, and video tokens through an integrated architecture combining Kimi Delta Attention (KDA), Multi-head Latent Attention (MLA), modality-aware local convolutions, and sparse Mixture-of-Experts (MoE). The study introduces HeteroP, a novel module-level heterogeneous scaling strategy that establishes, for the first time, Chinchilla-like compute-optimal laws governing active parameters, training token count, and image–video data ratios. Their 11B-parameter model activates only 2B parameters, achieving 1.7× higher compute efficiency in pretraining loss over dense baselines and 7.3× system-wide speedup. Notably, it enables zero-shot, no-finetuning extrapolation to 30-second videos, with FID degrading by merely 6.5% in the final five seconds.

Abstract

Visual generation increasingly requires high-resolution images, long videos, and multimodal context, making the quadratic cost of full attention prohibitive. We introduce Chimera, a hybrid visual diffusion backbone with a principled scaling recipe. Chimera processes text, image, and video tokens in one raster-ordered stream without positional embeddings. It combines Kimi Delta Attention (KDA) for long-context state tracking with O(N) complexity, interleaved Multi-head Latent Attention (MLA) for direct global interaction, and modality-aware short convolutions for local spatiotemporal context. Sparse Mixture-of-Experts (MoE) layers expand capacity while controlling activated compute. To scale this heterogeneous architecture, we introduce HeteroP, a module-wise scheme that transfers hyperparameters across width and depth according to each tensor's functional fan-in and model depth. HeteroP yields a consistently tuned family used to fit Chinchilla-style compute-optimal laws for activated model size, training-token count, and image-video data ratio. Guided by these laws, we train an 11B-parameter Chimera with 2B activated parameters. Experiments show three results. First, measured by pretraining diffusion loss, the dense backbone is 1.7x as compute-efficient as a matched full-attention Wan-2.1 2B baseline, while the complete system reaches 7.3x. Second, without length-specific fine-tuning, Chimera extrapolates zero-shot from 5-second training clips to 30-second videos, with only 6.5% FID degradation in the last five seconds. Third, the fitted laws show that compute-optimal image pretraining divides compute nearly evenly between activated model size and training-token count, whereas video pretraining modestly favors model size at higher budgets. These results establish a foundation for designing and scaling efficient long-context diffusion architectures.

Problem

• visual generation
• long-context modeling
• diffusion transformers
• computational efficiency
• quadratic attention cost

Innovation

• Hybrid Visual Diffusion Transformer
• Kimi Delta Attention
• HeteroP Scaling
• Sparse Mixture-of-Experts
• Compute-Optimal Scaling