PAST: Prompt-Adaptive Sampling Termination for Efficient Diffusion Model

2026-08-07

AI Summary

This work addresses the challenge of efficiently optimizing downstream objectives in text-to-image generation with diffusion models, which is hindered by the low sample efficiency of reinforcement learning fine-tuning and sparse extrinsic rewards. To overcome these limitations, the authors propose an adaptive reinforcement learning framework that dynamically terminates sampling episodes based on a joint assessment of denoising progress and prompt difficulty. Furthermore, they introduce an intrinsic reward mechanism grounded in semantic alignment and residual noise prediction to alleviate reward sparsity and guide the model toward high-quality generation trajectories. Experimental results demonstrate that the proposed approach substantially improves both training efficiency and generation quality, achieving up to a 66.7% gain in computational efficiency and a 29.5% improvement in preference-aligned output quality.

Abstract

While diffusion models have made significant progress in text-to-image tasks, they still exhibit limitations when directly optimizing downstream objectives. Although Reinforcement Learning (RL) enables targeted optimization, existing methods are generally constrained by low-efficiency fine-tuning and sparse rewards. To address these challenges, we propose PAST, which provides differentiated rewards while adaptively regulating training episode length by jointly perceiving denoising progress and prompt difficulty. Specifically, we design an intrinsic reward paradigm to compensate for sparse extrinsic rewards and guide the model to explore paths that diverge more efficiently from noise patterns. We further provide theoretical justification for intrinsic rewards. Then, PAST dynamically monitors denoising completion and semantic alignment between image structures and prompt semantics. When both metrics satisfy generation requirements, the system adaptively terminates training. This enables appropriate allocation of episode lengths based on prompt difficulty and the current generation process. Finally, based on the predicted residual noise level, we establish a dual adaptive coordination mechanism. Specifically, it not only balances the extrinsic and intrinsic rewards but also balances the exploration and convergence. Experimental results demonstrate that PAST enhances computational efficiency of existing RL fine-tuning methods by up to 66.7%, while improving preference optimization quality by up to 29.5% through its dual adaptive regulation mechanism.

Problem

• diffusion models
• reinforcement learning
• sparse rewards
• fine-tuning efficiency
• text-to-image generation

Innovation

• Prompt-Adaptive Sampling Termination
• Intrinsic Reward
• Diffusion Model
• Reinforcement Learning Fine-tuning
• Adaptive Episode Termination