Unified Video Dense Prediction from Disjoint Data

2026-07-23

AI Summary

This work addresses the challenge that existing scene understanding methods are hindered by task-specific and incompatible datasets, making it difficult to jointly model multi-dimensional dense prediction tasks such as geometry, appearance, and semantics. The paper proposes UniD, the first framework that enables unified video dense prediction across heterogeneous domains without requiring overlapping annotations or pseudo-labels. UniD leverages a pretrained diffusion model to provide strong visual priors and employs lightweight task projectors to distill knowledge from multiple task-specific expert models into a single shared backbone. Evaluated on eight diverse tasks, UniD matches or surpasses both task-specific and multitask baselines while significantly improving temporal consistency, cross-task coherence, and generalization to unseen scene–task combinations.

Abstract

Scene understanding requires simultaneous prediction about geometry, appearance, and semantics. However, existing task-specific annotations are fragmented across incompatible, domain-specific datasets. Current unified systems circumvent this by restricting training to fully co-annotated data, or by incurring the large computational cost of pseudo-labeling. To mitigate this, we introduce UniD, a unified video model that jointly predicts eight dense scene properties-depth, surface normals, semantic segmentation, boundaries, human parts, albedo, shading, and materials-all learned from disjoint, domain-specific datasets. We propose a simple yet effective distillation step in which per-task experts supervise a unified backbone through lightweight task projectors, eliminating the need for annotation overlap or pseudo-labeling. Our key insight is that the strong visual priors of a pretrained diffusion model are sufficient to bridge the domain gaps introduced by disjoint training sources, enabling robust generalization to scene-task combinations never seen during training. UniD achieves competitive performance against per-task specialists and multi-task baselines, with strong generalization to out-of-distribution scenarios and enhanced temporal and cross-task consistency. Code and video results are available at https://unid-video.github.io/.

Problem

• dense prediction
• disjoint datasets
• unified video model
• scene understanding
• multi-task learning

Innovation

• unified video model
• disjoint datasets
• task distillation
• diffusion priors
• dense prediction