GraphVid: Interactive Graph-Controllable Video Generation

2026-07-23

AI Summary

This work addresses the limitations of existing controllable video generation methods, which struggle to accurately model object interactions in complex scenes with multiple objects or occlusions and are often constrained by trajectory-based control that restricts expressive capacity. To overcome these challenges, the authors propose a novel image-to-video generation framework conditioned on structured interaction graphs, introducing a new paradigm where such graphs serve as explicit control signals. They further contribute GraphVid-Bench, the first large-scale video dataset annotated with structured relational information. By integrating a graph-conditioned control mechanism into an image-to-video diffusion architecture, the method enables flexible and precise manipulation of multi-agent motion and relationships through a structured semantic interface. Experiments demonstrate significant improvements over Motion-I2V, with FID reduced by 39.9%, FVD by 37.6%, PSNR increased from 9.87 to 15.98, and SSIM from 0.38 to 0.61.

Abstract

Controllable video generation remains challenging due to the difficulty of specifying precise multi-object interactions using text prompts or motion-control inputs that primarily constrain pixel movement. In practice, trajectory-based control often requires users to draw accurate tracks for multiple objects, which scales poorly with scene complexity and becomes ambiguous under occlusion or overlap. To enable flexible yet precise multi-subject control, we introduce $\textbf{GraphVid}$, a graph-conditioned image-to-video generation model that enables interactive control through structured interaction graphs. We further curate $\textbf{GraphVid-Bench}$, a large-scale interaction-centric video dataset with structured relational annotations to enable training of interaction-aware video generation models. Despite using substantially less training data and fewer trainable parameters than prior motion-control methods, GraphVid delivers strong controllability and video quality. Compared with Motion-I2V, GraphVid reduces FID by up to 39.9% and FVD by 37.6%, while improving PSNR (9.87=>15.98) and SSIM (0.38=>0.61). Our results highlight the potential of structured semantic interfaces as a powerful paradigm for controllable video generation.

Problem

• controllable video generation
• multi-object interaction
• trajectory-based control
• occlusion
• structured semantic interfaces

Innovation

• graph-conditioned generation
• interactive control
• multi-object interaction
• structured semantic interface
• controllable video generation