MagicPrompt: Ultra-Lightweight Prompt Tuning for Video Generation

2026-07-16

AI Summary

This work addresses the high computational cost and instability associated with fine-tuning large-scale video diffusion models for conditional generation tasks. Existing parameter-efficient methods still require a substantial number of trainable parameters, while reward-based optimization often suffers from training instability. To overcome these limitations, the authors propose MagicPrompt, a novel framework that synergistically integrates Attention-Embedded Prompt Tuning with Dual-Space Reward Feedback Optimization. By leveraging this approach, MagicPrompt achieves significantly improved training stability and generation quality while preserving pre-trained knowledge, all with fewer than 1% trainable parameters. Consequently, the method substantially reduces computational overhead without compromising performance.

Abstract

Large-scale video diffusion models (VDMs) deliver strong generation performance, but full fine-tuning for downstream tasks incurs prohibitive computational costs. Existing parameter-efficient fine-tuning (PEFT) methods have two critical flaws on billion-scale models: they still require substantial trainable parameters, and reward-based training suffers from noise-induced optimization instability in condition-guided tasks. We propose MagicPrompt, a lightweight framework that achieves extreme parameter efficiency and stable reward optimization. It first adopts Attention-Embedded Prompt Tuning, which steers generation via lightweight soft prompts with orders of magnitude fewer parameters while preserving pre-trained knowledge. It further introduces Dual-Space Reward Feedback Optimization, which uses self-supervised latent objectives to improve condition-guided reward training. Experiments show MagicPrompt reaches competitive performance with less than 1\% trainable parameters and notably reduces training costs.

Problem

• video diffusion models
• parameter-efficient fine-tuning
• reward optimization
• computational cost
• condition-guided generation

Innovation

• Prompt Tuning
• Parameter-Efficient Fine-Tuning
• Video Diffusion Models
• Reward Optimization
• Attention-Embedded Prompts