Vorch-Omni: Multi-Task Orchestration of Sight and Sound

2026-08-06

AI Summary

This work proposes a unified multitask audiovisual synthesis framework that overcomes the limitations of existing task-specific models, which struggle to handle diverse input–output configurations and suffer from cross-task interference. Built upon an “any-to-any” conditional generation paradigm, the framework leverages masking mechanisms, task identifiers, and positional encodings to disambiguate signal roles within a single architecture, enabling support for over ten audiovisual generation and editing tasks—including text-to-video/audio, reference-guided synthesis, temporal extension, and audio-driven generation—without requiring task-specific architectural modifications. The model integrates flow-matching diffusion Transformers, vision-language models, video VAEs, and a distributed data pipeline, and introduces dual visual conditioning pathways to enhance semantic and structural understanding, thereby establishing a scalable foundation model for general-purpose audiovisual synthesis.

Abstract

Recent advances in generative video modeling have enabled diverse generation, reference-based synthesis, extension, and editing, but existing approaches often rely on fragmented task-specific models. A general model must distinguish heterogeneous target, source, and reference signals to determine what to generate, preserve, or use as guidance, while reducing interference among tasks. Joint audio-visual generation further increases this challenge by introducing diverse conditioning and output configurations across modalities. We present Vorch-Omni, a unified multi-task framework for audio-visual synthesis based on an arbitrary-condition-to-arbitrary-output formulation. It flexibly treats video and audio signals as either conditioning inputs or generation targets. Token-level conditioning masks and task identifiers distinguish targets, source content, and references, while position types separate temporal context from independent conditions. To capture semantic and structural information, Vorch-Omni employs complementary visual conditioning pathways: a vision-language model interprets sampled frames with text instructions, and a video VAE encodes conditions into latent tokens for direct guidance. We further build a distributed data pipeline to curate diverse temporally aligned audio-visual clips, generate structured captions and metadata, and balance heterogeneous task distributions. Built on a single flow-matching diffusion transformer without task-specific architectural changes, Vorch-Omni supports over 10 tasks, including text-to-video, text-to-audio-video, image- and reference-conditioned generation, temporal extension, audio-driven generation, video transformation, and audio-visual editing. This unified framework provides a scalable foundation for general-purpose audio-visual generation and manipulation.

Problem

• multi-task learning
• audio-visual generation
• unified framework
• conditional generation
• task interference

Innovation

• multi-task audio-visual generation
• arbitrary-condition-to-arbitrary-output
• token-level conditioning masks
• complementary visual conditioning pathways
• flow-matching diffusion transformer