FestDPO: Few-step Generator Alignment with Direct Preference Optimization
This study addresses the challenge of aligning few-step generative models via Direct Preference Optimization (DPO), where likelihood evaluation is infeasible due to their implicit nature. To overcome this bottleneck, we propose FestDPO, a framework that leverages non-parametric likelihood estimation and rapid sampling capabilities to construct a sample-level DPO loss approximation independent of model architectures and sampling procedures. Experimental evaluations demonstrate that FestDPO surpasses baseline methods in both win rate and human ratings for text-to-image generation. Furthermore, in protein backbone generation, it significantly improves β-sheet proportions and structural designability. These results validate the generalizability and effectiveness of the proposed approach across diverse domains, establishing a principled pathway for preference alignment in few-step generative modeling.