rlhf training

This work systematically investigates reinforcement learning (RL)-driven alignment and capability enhancement of large language models (LLMs), addressing three core challenges: instruction following, ethical compliance, and complex reasoning. Methodologically, it introduces a two-dimensional classification framework grounded in reward modeling and policy optimization to unify the analysis of prominent paradigms—including RLHF, DPO, RLAIF, GRPO, and RLVR. Empirical analysis reveals emerging trends: RLHF excels at foundational alignment, while RLVR significantly improves stepwise reasoning. The study identifies critical bottlenecks—reward gaming, multi-objective trade-offs, and computational overhead—and proposes novel directions: hybrid RL architectures and verifier-guided training. Collectively, this work delivers a principled technical roadmap and methodological foundation for developing safe, reliable, and scalable RL-augmented LLMs.

To address the high cost of human feedback and low sample efficiency in reward function learning for human-in-the-loop reinforcement learning, this paper proposes the Suboptimal Data Pretraining (SDP) framework. SDP enables cold-start training of reward models without human annotations by leveraging unlabeled, low-quality trajectory data—augmented with pseudo-labels derived from environment-minimum rewards. The method integrates pseudo-labeling, scalar reward modeling, and preference-based learning within a human-in-the-loop RL architecture. Evaluated across diverse simulated robotic tasks, SDP achieves significant improvements over state-of-the-art methods: it attains comparable or superior performance while reducing human interaction counts by over 50%. Crucially, SDP is compatible with both simulated and real human teachers and, for the first time, enables efficient reward modeling without any manual annotation.

This paper addresses the fragmentation and weak theoretical foundations of Reinforcement Learning from Human Feedback (RLHF) in large language model alignment. We propose the first multi-stage collaborative optimization framework integrating economic incentive mechanisms, philosophical value reasoning, and optimal control theory. Methodologically, we systematically unify instruction tuning, Bradley–Terry reward modeling, Proximal Policy Optimization (PPO), Direct Preference Optimization (DPO), rejection sampling, and a structured human feedback protocol. Our contributions are threefold: (1) a modular, reproducible end-to-end RLHF practice guide; (2) clarification of key open challenges—including synthetic data generation and multi-dimensional alignment evaluation; and (3) enhanced model safety, controllability, and value consistency. The framework bridges rigorous theoretical grounding with practical engineering applicability, providing a principled methodology for deploying trustworthy large language models.

This paper addresses the narrow applicability and neglect of human factors in Reinforcement Learning from Human Feedback (RLHF). We propose the first systematic framework integrating human factors engineering, interaction design, and RL modeling requirements. Methodologically, we introduce a nine-dimensional taxonomy of human feedback and seven quality metrics to unify perspectives from human–computer interaction, interface design, and RL modeling; further, through conceptual modeling and cross-disciplinary analysis, we establish interdisciplinary design principles and identify critical research gaps. Our contribution is threefold: (1) the first scalable theoretical framework for feedback space representation; (2) foundational support for designing high-quality, human–machine collaborative adaptive learning systems; and (3) advancement of data-driven co-adaptive modeling and diverse interaction mechanisms—thereby enabling standardized, interpretable, and human-centered RLHF development. (149 words)

Current large language model training typically introduces reinforcement learning (RL) only after pretraining and supervised fine-tuning (SFT), which constrains its full potential. This work proposes a novel paradigm that integrates RL and SFT directly during multiple stages of pretraining, exploring their concurrent optimization. By intervening at pretraining checkpoints, designing a target objective averaging mechanism, and carefully controlling data composition, the study demonstrates that introducing RL early can match or even surpass the performance of the conventional SFT→RL pipeline—particularly on challenging tasks—without compromising general capabilities. Moreover, strategic design of data composition proves more effective for performance gains than merely scaling up model size. These findings offer a new, efficient, and flexible pathway for aligning language models with desired behaviors.

This study addresses the challenges of safety alignment, feedback quality, and bidirectional adaptation in Reinforcement Learning from Human Feedback (RLHF) within human–machine collaboration. It is the first to investigate the bidirectional closed-loop mechanism of RLHF. Methodologically, a systematic review was conducted following PRISMA guidelines, complemented by empirical analyses integrating virtual reality experiments with Bayesian modeling. The results demonstrate that feedback timing significantly influences interaction quality. Furthermore, compared to system-initiated prompts, user-initiated feedback more precisely captures psychological safety and enhances overall system security. By establishing both a theoretical foundation and empirical evidence, this work provides critical insights for optimizing RLHF frameworks in collaborative human–machine systems.

This work addresses the challenges of high noise, strong subjectivity, and heterogeneity in human feedback within reinforcement learning from human feedback (RLHF) by proposing, for the first time, a unified statistical framework that models its core components cohesively. It systematically connects supervised fine-tuning, reward modeling, and policy optimization with established statistical methods—namely the Bradley–Terry–Luce model, latent utility estimation, and active learning—thereby unifying two-stage and one-stage paradigms such as direct preference optimization. The framework further extends to emerging directions including AI-generated feedback and verifiable rewards. Integrating experimental design and uncertainty quantification, this study establishes a rigorous statistical foundation for RLHF, accompanied by open-source code and benchmark datasets to guide future methodological development and empirical research.

Existing LLM instruction-tuning algorithms—such as supervised fine-tuning (SFT), proximal policy optimization (PPO), and direct preference optimization (DPO)—are often explained with heavy reliance on prior knowledge, omit critical derivations, or remain overly abstract, resulting in high cognitive barriers and poor interpretability. Method: This paper systematically unifies mainstream reinforcement learning and preference optimization approaches under a concise, symbolically grounded derivation framework explicitly tailored to practical LLM training scenarios. Contribution/Results: We introduce GRAPE (Generalized Relative Advantage Policy Evolution), a novel paradigm for future preference learning designed to overcome fundamental limitations of current methods in objective design, training stability, and generalization. The framework provides a coherent, step-by-step exposition—from SFT through DPO—enhancing algorithmic intuition and theoretical transparency. It establishes a rigorous foundation for advancing preference-based LLM alignment and offers principled directions for subsequent research.

This work addresses the challenges of deploying reinforcement learning on real-world robotic systems, where inefficient and unsafe exploration hinders practical application, and existing approaches fail to effectively leverage preference information embedded in human interventions. To overcome these limitations, the authors propose a state-dependent adaptive preference gating mechanism that, for the first time, models human intervention as state-specific relative preferences rather than action demonstrations, dynamically modulating the influence of human feedback on policy learning. Integrating online preference learning, reinforcement learning optimization, and a gating architecture, the method is evaluated on a Franka robot across multiple contact-rich manipulation tasks. Experimental results demonstrate that, compared to baseline methods, the proposed approach significantly improves learning efficiency and safety, achieving higher task success rates, faster convergence, reduced human intervention, and more stable, human-aligned policy behavior.

This study addresses the inefficient training in human-in-the-loop reinforcement learning (HIL-RL) caused by underutilized human experience and imitation penalties that constrain policy optimization. To overcome these limitations, this work proposes the ReF-HIL framework, which innovatively integrates independent value reference learning with a dynamic human action neighborhood mechanism. Specifically, it introduces human-reference-guided value shaping to eliminate intrinsic imitation penalties and accelerate learning, while defining a human action fence to constrain extrinsic updates, thereby achieving an effective balance between autonomous exploration and human demonstration. Evaluated across five real-world robotic tasks, the proposed framework attains a 90% success rate within merely 18 to 63 minutes of training, ultimately reaching final success rates of 91.7% to 100%. These results demonstrate that ReF-HIL significantly enhances learning efficiency in HIL-RL settings.

为解决长时任务中关键子任务失败问题，提出PARTS框架，通过集中训练瓶颈子任务并结合在线RL和成功加权重训方法，减少人类干预。