build sft and rlhf pipelines

This work systematically investigates reinforcement learning (RL)-driven alignment and capability enhancement of large language models (LLMs), addressing three core challenges: instruction following, ethical compliance, and complex reasoning. Methodologically, it introduces a two-dimensional classification framework grounded in reward modeling and policy optimization to unify the analysis of prominent paradigms—including RLHF, DPO, RLAIF, GRPO, and RLVR. Empirical analysis reveals emerging trends: RLHF excels at foundational alignment, while RLVR significantly improves stepwise reasoning. The study identifies critical bottlenecks—reward gaming, multi-objective trade-offs, and computational overhead—and proposes novel directions: hybrid RL architectures and verifier-guided training. Collectively, this work delivers a principled technical roadmap and methodological foundation for developing safe, reliable, and scalable RL-augmented LLMs.

Current large language model training typically introduces reinforcement learning (RL) only after pretraining and supervised fine-tuning (SFT), which constrains its full potential. This work proposes a novel paradigm that integrates RL and SFT directly during multiple stages of pretraining, exploring their concurrent optimization. By intervening at pretraining checkpoints, designing a target objective averaging mechanism, and carefully controlling data composition, the study demonstrates that introducing RL early can match or even surpass the performance of the conventional SFT→RL pipeline—particularly on challenging tasks—without compromising general capabilities. Moreover, strategic design of data composition proves more effective for performance gains than merely scaling up model size. These findings offer a new, efficient, and flexible pathway for aligning language models with desired behaviors.

为解决长时任务中关键子任务失败问题，提出PARTS框架，通过集中训练瓶颈子任务并结合在线RL和成功加权重训方法，减少人类干预。

This work addresses the need for post-training to enhance the accuracy and reasoning reliability of large language models on specific tasks, while the boundaries and synergies between supervised fine-tuning (SFT) and reinforcement learning (RL) remain unclear. The study proposes a unified analytical framework to systematically compare SFT and RL in terms of objective formulation, algorithmic structure, and data requirements, revealing their intrinsic connections. Building on this analysis, the authors design an integrated strategy to establish an efficient hybrid post-training paradigm. Through empirical evaluation across representative applications from 2023 to 2025, the research identifies a clear trend toward hybrid post-training approaches and distills key practical guidelines, offering both theoretical grounding and methodological guidance for scalable, effective, and generalizable post-training of large language models.

This paper addresses the fragmentation and weak theoretical foundations of Reinforcement Learning from Human Feedback (RLHF) in large language model alignment. We propose the first multi-stage collaborative optimization framework integrating economic incentive mechanisms, philosophical value reasoning, and optimal control theory. Methodologically, we systematically unify instruction tuning, Bradley–Terry reward modeling, Proximal Policy Optimization (PPO), Direct Preference Optimization (DPO), rejection sampling, and a structured human feedback protocol. Our contributions are threefold: (1) a modular, reproducible end-to-end RLHF practice guide; (2) clarification of key open challenges—including synthetic data generation and multi-dimensional alignment evaluation; and (3) enhanced model safety, controllability, and value consistency. The framework bridges rigorous theoretical grounding with practical engineering applicability, providing a principled methodology for deploying trustworthy large language models.

Existing RLHF frameworks suffer from inference bottlenecks, deployment complexity, challenges in multi-model orchestration, and low resource utilization—hindering the accessibility and scalable deployment of large language model (LLM) alignment. To address these limitations, this paper introduces the first open-source RLHF training framework specifically designed for LLM alignment. It features a cross-GPU heterogeneous scheduling architecture that decouples the reward model, policy model, reference model, and value model for independent deployment. The framework natively supports multiple alignment paradigms—including RLHF, DPO, and rejection sampling—within a unified interface. Leveraging Ray for elastic task orchestration, it tightly integrates vLLM (for high-throughput inference) and DeepSpeed (for efficient training), while maintaining native compatibility with the Hugging Face ecosystem. Experiments demonstrate substantial improvements in training throughput and GPU memory efficiency for models ≥70B parameters. The framework delivers an out-of-the-box, end-to-end alignment solution and has been open-sourced, gaining broad adoption in the research and engineering community.

本文提出TailSFT方法，通过过滤已拟合序列来改进监督微调，从而提高模型在强化学习中的表现。

This work addresses the challenges of high noise, strong subjectivity, and heterogeneity in human feedback within reinforcement learning from human feedback (RLHF) by proposing, for the first time, a unified statistical framework that models its core components cohesively. It systematically connects supervised fine-tuning, reward modeling, and policy optimization with established statistical methods—namely the Bradley–Terry–Luce model, latent utility estimation, and active learning—thereby unifying two-stage and one-stage paradigms such as direct preference optimization. The framework further extends to emerging directions including AI-generated feedback and verifiable rewards. Integrating experimental design and uncertainty quantification, this study establishes a rigorous statistical foundation for RLHF, accompanied by open-source code and benchmark datasets to guide future methodological development and empirical research.

This work addresses the long-standing isolation among research domains such as alignment training, model organisms, and toy models, which has hindered empirical cross-pollination and led to redundant exploration and inefficiency. For the first time, it systematically transfers supervised fine-tuning (SFT) practices across these domains by integrating cross-model output training, mixed-strategy data, and benign fine-tuning to rigorously evaluate the portability of key findings. The study demonstrates three successful transfer effects: enhanced behavioral generalization, mitigation of capability degradation, and the critical insight that preserving capabilities alone is insufficient to ensure robustness in subsequent training phases. These results underscore both the efficacy and limitations of reusing methodologies across domains, thereby fostering more synergistic development across disparate research areas.

This work addresses the lack of a general, auditable dynamic control mechanism in existing training systems, which typically rely on framework-specific code. The authors propose the first cross-framework, open-source control plane that exposes training interfaces through a unified protocol, integrating declarative configuration, request validation, and secure control-point scheduling within the Aim workspace to enable metric monitoring, real-time intervention, and operational traceability. The system supports safe human and automated controller interventions during training while fully logging all operational trajectories. Experiments across five NLP and reinforcement learning tasks demonstrate its effectiveness, and the open-source implementation provides a foundation for reproducible human-in-the-loop training.

This study addresses the limitations of supervised fine-tuning (SFT), which suffers from weak generalization and catastrophic forgetting due to its reliance on off-policy data. To overcome these bottlenecks, this work proposes a Markov Chain Monte Carlo (MCMC)-based sampling algorithm that formulates sampling as a model-native operator for the first time. Guided by a reference model, the approach progressively transforms off-policy expert data into an on-policy distribution. By reshaping the data distribution rather than modifying the objective function, it enables SFT to effectively leverage privileged information. Experiments on scientific skill acquisition and mathematical reasoning tasks demonstrate that the proposed method allows SFT to achieve performance comparable to reinforcement learning, while significantly mitigating catastrophic forgetting and enhancing out-of-distribution generalization capabilities.