SPEED: One-Step Pixel Diffusion for High-quality Video Frame Interpolation

2026-07-16

AI Summary

This work addresses the high memory consumption, inference latency, and detail degradation inherent in existing diffusion-based video frame interpolation methods that rely on multi-step sampling. To overcome these limitations, we propose SPEED—a single-step, pixel-level diffusion framework that jointly models multi-scale motion, structure, and appearance through a progressive multi-stage architecture with dynamic block scaling, directly predicting intermediate frames in pixel space. Key innovations include a Noise-Update-Only Attention mechanism that preserves semantic fidelity of conditioning frames while reducing computational overhead by nearly 50%, and a Drift-aware Timestep Sampling strategy that enhances single-step generation quality. Experiments demonstrate that SPEED achieves an 8.8% lower LPIPS on SNU-FILM, 63.3% faster inference, and 10.6% less memory usage; on 4K benchmarks, it improves LPIPS by up to 51.5% over prior methods.

Abstract

Despite the success of diffusion models in Video Frame Interpolation (VFI), existing methods still suffer from two critical limitations. First, latent diffusion inevitably loses fine-grained details when reconstructing images from latent representations back to the pixel space. Second, multi-step sampling incurs prohibitive memory consumption and inference latency. To address these issues, we propose SPEED, a one-step pixel diffusion framework for high-quality VFI. Specifically, SPEED employs a progressive multi-stage architecture with dynamic patch scaling to effectively learn multi-scale motion, structural, and appearance representations. Furthermore, we propose a novel Noise-Update-Only Attention mechanism to prevent semantic degradation of the clean condition frames while reducing the computational overhead by nearly 50%. Besides, we introduce a Drift-aware Timestep Sampling strategy coupled with a tailored training objective to directly predict images in the pixel space, enabling one-step inference without compromising the quality of the generated frames. Extensive experiments show that SPEED achieves state-of-the-art performance. On SNU-FILM, SPEED reduces LPIPS by 8.8% while delivering 63.3% faster inference and 10.6% lower memory usage. On challenging 4K benchmarks, it further surpasses prior methods by up to 51.5% in LPIPS.

Problem

• Video Frame Interpolation
• Diffusion Models
• Pixel-space Reconstruction
• Multi-step Sampling
• Fine-grained Details

Innovation

• one-step diffusion
• pixel-space generation
• Noise-Update-Only Attention
• Drift-aware Timestep Sampling
• video frame interpolation