AdaGRPO: A Capability-Aware Adaptive Enhancement for Flow-based GRPO

2026-06-04

AI Summary

This work addresses the inefficiency and instability of existing flow-based GRPO methods, which decouple prompt selection and advantage estimation from the model’s current capabilities. To overcome these limitations, we propose AdaGRPO, a capability-aware reinforcement learning algorithm that dynamically aligns training prompts with the model’s evolving learning frontier through an online curriculum-based prompt filtering mechanism. AdaGRPO further introduces a hybrid advantage estimator that combines intra-group and global advantages to enable unbiased policy evaluation. Designed as a lightweight plug-in, AdaGRPO seamlessly integrates with mainstream frameworks such as Flow-GRPO and DanceGRPO, significantly improving alignment performance in text-to-image generation while enhancing training stability and convergence efficiency.

Abstract

Group Relative Policy Optimization (GRPO) has demonstrated remarkable success in aligning text-to-image (T2I) flow models with human preferences. However, we have identified that the learning loop of current flow-based GRPO is fundamentally decoupled from the learner's current capability, suffering from critical blind spots at both prompt selection and advantage estimation: (i) Existing methods sample prompts randomly, overlooking the substantial impact of data selection on reinforcement learning (RL) efficacy--a factor proven crucial in GRPO for large language models; (ii) They evaluate sample quality solely relying on intra-group statistics, lacking a global perspective to accurately measure true policy improvement. To address these issues, we propose Adaptive GRPO (AdaGRPO), a novel capability-aware RL algorithm tailored for flow models. Specifically, AdaGRPO consists of two principal components: (i) Online Curriculum Filtering Strategy: Dynamically tracks the model's proficiency and adaptively selects prompts that best match its current learning boundary; (ii) Cross-Level Advantage Fusion: Synergistically integrates fine-grained intra-group advantages with macro-level global advantages, providing a comprehensive and unbiased policy evaluation. As a lightweight, plug-and-play module, AdaGRPO can be seamlessly integrated with existing frameworks such as Flow-GRPO, DanceGRPO, and Flow-CPS. Extensive experiments demonstrate that AdaGRPO consistently drives performance gains while significantly stabilizes GRPO training for flow models.

Problem

• flow-based GRPO
• prompt selection
• advantage estimation
• reinforcement learning
• text-to-image alignment

Innovation

• Adaptive GRPO
• Online Curriculum Filtering
• Cross-Level Advantage Fusion
• Flow-based Reinforcement Learning
• Capability-Aware Policy Optimization