🤖 AI Summary
Existing text-to-video (T2V) models struggle to generate long-duration, dynamically evolving videos, suffering from static content generation, temporal forgetting, and severe computational bottlenecks when scaling to extended sequences. To address these limitations, we propose Generative Temporal Nurturing (GTN), a novel inference-time paradigm that steers the diffusion process toward precise spatiotemporal evolution of visual states. GTN introduces two lightweight, plug-and-play components: Video Summary Prompting (VSP), which leverages large language models to automatically generate segmented semantic summaries; and Temporal Attention Regularization (TAR), a parameter-efficient module compatible with off-the-shelf T2V models (e.g., AnimateDiff) without retraining. Evaluated across multiple benchmarks, GTN significantly improves video length, motion dynamics, and semantic fidelity—producing longer, more coherent videos whose temporal evolution aligns closely with textual descriptions. Visual ablation studies confirm GTN’s effectiveness in mitigating temporal forgetting.
📝 Abstract
Despite tremendous progress in the field of text-to-video (T2V) synthesis, open-sourced T2V diffusion models struggle to generate longer videos with dynamically varying and evolving content. They tend to synthesize quasi-static videos, ignoring the necessary visual change-over-time implied in the text prompt. At the same time, scaling these models to enable longer, more dynamic video synthesis often remains computationally intractable. To address this challenge, we introduce the concept of Generative Temporal Nursing (GTN), where we aim to alter the generative process on the fly during inference to improve control over the temporal dynamics and enable generation of longer videos. We propose a method for GTN, dubbed VSTAR, which consists of two key ingredients: 1) Video Synopsis Prompting (VSP) - automatic generation of a video synopsis based on the original single prompt leveraging LLMs, which gives accurate textual guidance to different visual states of longer videos, and 2) Temporal Attention Regularization (TAR) - a regularization technique to refine the temporal attention units of the pre-trained T2V diffusion models, which enables control over the video dynamics. We experimentally showcase the superiority of the proposed approach in generating longer, visually appealing videos over existing open-sourced T2V models. We additionally analyze the temporal attention maps realized with and without VSTAR, demonstrating the importance of applying our method to mitigate neglect of the desired visual change over time.