OmniVR: Joint Video-Audio Conditional Generation for Restoring Degraded Historical Films

2026-08-04

AI Summary

Historical films often suffer from concurrent visual and audio degradations—such as blur, noise, flicker, and acoustic artifacts—and existing methods typically address each modality in isolation, resulting in suboptimal restoration quality and cross-modal inconsistencies. This work proposes OmniVR, the first unified generative model for joint audiovisual restoration, formulating the task as conditional generation within a multimodal Diffusion Transformer (DiT). By jointly denoising latent representations of degraded audiovisual inputs guided by contextual prompts, OmniVR simultaneously recovers visual structure, temporal dynamics, and acoustic fidelity. Key contributions include a novel joint audiovisual restoration framework, a realistic degradation pipeline mimicking authentic archival film deterioration, architecture-preserving transfer from text-to-audiovisual (T2AV) to audiovisual-to-audiovisual (AV2AV) settings, and techniques such as first-frame anchoring and waveform-level supervision to enable long-sequence video extrapolation and high-fidelity audio reconstruction. Evaluated on the newly introduced OmniVRBench benchmark—comprising 200 real historical clips—and powered by a 22-billion-parameter backbone, OmniVR achieves state-of-the-art performance across six visual metrics and superior audio quality, while demonstrating the first synchronized high-quality colorization, video restoration, and audio recovery.

Abstract

Historical films suffer from co-occurring visual and audio degradations---blur, noise, flicker, hiss, clipping, and dropout---yet existing methods restore each modality independently, leaving quality gaps and cross-modal inconsistency. We present OmniVR, the first joint audio-video generative restoration model. Built upon a 22B-parameter audio-video generation backbone, OmniVR formulates restoration as conditional generation within a unified multimodal DiT: the low-quality video and audio are encoded as latent conditions, combined with a fixed restoration prompt, and jointly denoised to recover visual structure, temporal motion, and acoustic detail under one coordinated objective. Three key designs enable this adaptation: (1) a joint audio-video degradation pipeline that simulates real old-film characteristics from Internet-collected data; (2) an architecture-preserving text-to-audio-video (T2AV) to audio-video-to-audio-video (AV2AV) transition with prompt annealing that maximally retains the generative prior; and (3) first-frame image-to-video (I2V) anchoring with loss reweighting and waveform supervision for long-video extrapolation and audio fidelity. We also propose OmniVRBench, the first benchmark that evaluates audio-video restoration across visual quality, audio quality, temporal consistency, and audio-visual synchrony on 200 real historical clips. OmniVR surpasses all prior methods on all six visual metrics, achieves the best audio quality, and produces natural colorization---the first method to jointly address all three aspects. Code and weights will be publicly released. Project Page: https://xin1u.github.io/OminiVR_PAGE/

Problem

• historical film restoration
• audio-visual degradation
• multimodal inconsistency
• joint restoration
• conditional generation

Innovation

• joint audio-video restoration
• conditional generative model
• multimodal DiT
• degradation simulation pipeline
• prompt annealing