CAPE-T2V: Captioner-Anchored Prompt Enhancement toward Two-Sided Conditioning Alignment in Text-to-Video Generation

2026-08-03

AI Summary

This work addresses the residual mismatch—termed the PE-Caption gap—between prompt enhancer outputs and training captions in text-to-video generation, which arises from discrepancies in detail selection, information structuring, descriptive granularity, and phrasing. To bridge this gap, the authors propose a two-stage alignment framework: first, they fine-tune the prompt enhancer using three types of sample pairs to align its outputs with captions generated by a captioner; then, they employ the enhanced prompts to rewrite video captions for fine-tuning a diffusion Transformer, while uniformly processing user prompts during inference to achieve bidirectional semantic alignment between training and inference conditions. This study is the first to formally define and quantify the PE-Caption gap, introducing a captioner-anchored prompt enhancement strategy coupled with bidirectional fine-tuning, which significantly improves composite scores on StoryEval, VBench-2.0, and T2V-CompBench benchmarks for Wan2.2 and LTX-2.3 models, effectively narrowing the semantic distribution gap.

Abstract

Text-to-video (T2V) diffusion transformers (DiTs) are trained with detailed video captions, whereas inference often relies on user prompts rewritten by a prompt enhancer (PE). Prior work has improved generation by optimizing the PE, the DiT, or both; some methods have also sought to narrow the training-inference mismatch through shared schemas. Yet even within a shared schema, inference-time PE outputs and DiT training captions may still differ in detail selection, information organization, descriptive granularity, and phrasing. We refer to this residual mismatch as the PE-Caption gap and introduce CAPE-T2V, a two-step Captioner-Anchored Prompt Enhancement framework toward two-sided conditioning alignment in T2V generation. First, CAPE-T2V constructs three types of PE training examples, pairing captioner-generated targets with concise source captions, detailed source captions, or pseudo user prompts derived from those targets. It then fine-tunes the PE to map each input to its paired target. Second, CAPE-T2V fine-tunes the DiT on video-derived captions rewritten by the Anchored PE; the same PE rewrites user prompts at inference. Relative to a baseline using the same caption schema, CAPE-T2V achieves higher aggregate scores on StoryEval, VBench-2.0, and T2V-CompBench across Wan2.2 and LTX-2.3. Further, CAPE-T2V exhibits a smaller PE-Caption gap than the baseline: its DiT fine-tuning captions are closer in distribution to inference-time PE outputs, as measured by squared maximum mean discrepancy in a fixed embedding space. Overall, these results support CAPE-T2V as an effective approach to mitigating the PE-Caption gap. The project is available at https://github.com/yizzz927/CAPE-T2V.

Problem

• text-to-video generation
• prompt enhancement
• training-inference mismatch
• conditioning alignment
• caption-prompt gap

Innovation

• Prompt Enhancement
• Conditioning Alignment
• Text-to-Video Generation
• Diffusion Transformers
• Captioner-Anchored Training