JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion

2026-08-04

AI Summary

This work addresses the challenges of real-time open-domain video editing under strict computational constraints, where maintaining source fidelity, long-term temporal consistency, and causality—without access to future frames—is critical. The paper introduces the first streaming editing framework based on a 16-billion-parameter autoregressive diffusion model. It mitigates training-inference mismatch, source distortion, and temporal drift through chunk-wise autoregressive adaptation, Source-Anchored Distribution Matching Distillation (SA-DMD), and long-range autoregressive distillation. The method achieves end-to-end 720p video editing at approximately 30 FPS on a single NVIDIA B200 GPU, significantly outperforming existing streaming approaches in both automatic and human evaluations while matching the performance of powerful offline systems.

Abstract

Real-time video editing requires low-latency causal generation with bounded computational resources while preserving source fidelity and long-term temporal consistency. We present JoyAI-Video-Edit, a 16B-parameter autoregressive diffusion framework for real-time, open-ended video editing without access to future frames or a predefined video duration. Our method combines chunk-wise autoregressive adaptation, Source-Anchored Distribution Matching Distillation (SA-DMD), and Long-Horizon Autoregressive Distillation to reduce train--inference mismatch, preserve source fidelity during two-step generation, and mitigate accumulated temporal drift. Extensive automatic and human evaluations show that JoyAI-Video-Edit substantially outperforms existing streaming editors and remains competitive with strong offline systems on both short and long videos. The complete system achieves end-to-end 720p video editing at approximately 30 FPS on a single Nvidia B200 GPU. Code is available at https://github.com/jd-opensource/JoyAI-Video-Edit.

Problem

• real-time video editing
• temporal consistency
• source fidelity
• open-ended editing
• low-latency generation

Innovation

• autoregressive diffusion
• real-time video editing
• temporal consistency
• distribution matching distillation
• streaming video generation