FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable Reasoning

2025-10-26

AI Summary

In the RLVR paradigm, large language models (LLMs) often erroneously internalize “flawed-positive trajectories”—those containing erroneous patterns such as unfounded guessing or logical leaps—as reliable reasoning strategies, thereby undermining reasoning fidelity and long-term capability growth. To address this, we propose a phased optimization framework: leveraging flawed-but-effective trajectories for accelerated early convergence, then progressively transitioning to high-quality reasoning in later stages. We introduce a parameter-free reward penalty mechanism tightly coupled with a generative reward model (GenRM), enabling dynamic identification of flawed-positive rollouts and fine-grained, process-level error localization. Our method incurs no additional token overhead and significantly improves answer correctness, reasoning reliability, and training stability across diverse tasks—including mathematical reasoning and code generation—marking the first approach to enable *controlled utilization* of defective positive signals.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Reinforcement learning with verifiable rewards (RLVR) has emerged as a promising paradigm for enhancing the reasoning capabilities of large language models (LLMs). In this context, models explore reasoning trajectories and exploit rollouts with correct answers as positive signals for policy optimization. However, these rollouts might involve flawed patterns such as answer-guessing and jump-in-reasoning. Such flawed-positive rollouts are rewarded identically to fully correct ones, causing policy models to internalize these unreliable reasoning patterns. In this work, we first conduct a systematic study of flawed-positive rollouts in RL and find that they enable rapid capability gains during the early optimization stage, while constraining reasoning capability later by reinforcing unreliable patterns. Building on these insights, we propose Flawed-Aware Policy Optimization (FAPO), which presents a parameter-free reward penalty for flawed-positive rollouts, enabling the policy to leverage them as useful shortcuts in the warm-up stage, securing stable early gains, while gradually shifting optimization toward reliable reasoning in the later refinement stage. To accurately and comprehensively detect flawed-positive rollouts, we introduce a generative reward model (GenRM) with a process-level reward that precisely localizes reasoning errors. Experiments show that FAPO is effective in broad domains, improving outcome correctness, process reliability, and training stability without increasing the token budget.

Problem

• Identifies flawed reasoning patterns in reinforcement learning rollouts
• Proposes penalty mechanism to balance shortcut usage and reliability
• Introduces generative model for precise localization of reasoning errors

Innovation

• Penalizes flawed-positive rollouts with parameter-free reward
• Uses generative reward model for process-level error detection
• Enables stable gains and reliable reasoning in two stages