self-play training

Traditional self-play frameworks assume agent symmetry, making them ill-suited for the inherent task and capability asymmetries in open-ended learning. To address this, we propose the Heterogeneous Adversarial Learning framework (HAP), which formalizes asymmetric teaching as a bidirectional min-max optimization process—enabling automatic curriculum generation without predefined task hierarchies. HAP dynamically synthesizes learner-adaptive task sequences via a teacher-student adversarial mechanism, real-time performance feedback modulation, and co-evolution of strategies. Extensive experiments across multiple domains demonstrate that HAP significantly accelerates learning convergence and improves final performance, achieving state-of-the-art results. Moreover, HAP exhibits strong generalization across both artificial agents and human learners, underscoring its broad applicability in adaptive educational and reinforcement learning settings.

This paper addresses two-player, perfect-information games without domain-specific priors (e.g., Hex) by proposing an end-to-end self-play reinforcement learning framework for learning high-quality state evaluation functions. Methodologically: (i) a tree-guided generalization mechanism enhances policy generalization; (ii) an unbounded-depth minimax expansion overcomes conventional depth limitations; (iii) safe-action-prioritized search and a novel probabilistic action selection distribution are introduced; and (iv) a multi-dimensional reinforcement heuristic reward—incorporating fast-win/slow-loss bias, mobility, and other strategic factors—is designed. Empirically, the approach significantly outperforms Mohex 2.0 on 11×11 and 13×13 Hex boards. Under the no-swap rule on 11×11 Hex, it achieves performance comparable to Mohex 3HNN—a human-knowledge-enhanced system—demonstrating that strong game-playing intelligence can be attained without human prior knowledge or handcrafted features.

Large language models (LLMs) exhibit limited high-level strategic planning capabilities in complex multi-agent games, while conventional reinforcement learning (RL) approaches suffer from heavy reliance on extensive training data. Method: This paper proposes a novel two-level tree search–driven LLM self-play learning paradigm that synergistically integrates Monte Carlo Tree Search (MCTS) with LLM-based reflective reasoning: strategic-level state evaluation and policy planning, and execution-level action generation and dialogue synthesis. End-to-end strategic skill acquisition is achieved via self-play and reinforcement feedback. Contribution/Results: To our knowledge, this is the first framework to bridge the gap between LLMs and symbolic decision-making. It significantly improves win rates and strategic robustness on benchmark games—including GOPS and *The Resistance: Avalon*—outperforming both standard RL baselines and state-of-the-art LLM skill-learning methods.

Current reinforcement learning (RL) frameworks lack native, non-intrusive support for curriculum learning (CL), requiring invasive code modifications to implement. To address this, we propose CLib—the first lightweight, general-purpose curriculum learning library. CLib features a unified API and modular architecture comprising: (i) an environment-agnostic curriculum scheduler, (ii) a distributed sampling adapter, and (iii) a cross-framework bridging layer supporting both PyTorch and TensorFlow backends. It integrates seamlessly with five+ mainstream RL libraries—including Ray RLlib and CleanRL—without altering underlying training logic. We demonstrate the first successful application of CL in complex environments NetHack and Neural MMO, and validate CLib across nine benchmark tasks, consistently outperforming state-of-the-art baselines. By eliminating implementation barriers, CLib lowers the entry threshold for CL adoption, promotes standardization, and enhances reproducibility in RL research.

Existing meta-learning methods suffer from limited generalizability, often being confined to specific algorithms or requiring differentiability assumptions. This paper proposes a general reinforcement learning–driven meta-learning framework that trains a teacher policy to dynamically guide arbitrary student algorithms—without imposing structural or differentiability constraints on the student. Key contributions include: (i) the first unified pedagogical paradigm for meta-learning; (ii) a parameter-behavior encoder that implicitly infers the student’s internal parameter state from its input-output behavior; and (iii) a reward function grounded in learning progress. Experiments across supervised and reinforcement learning tasks demonstrate that our framework significantly outperforms baselines relying on heuristic rewards and handcrafted state representations, validating its broad generalizability and empirical effectiveness.

Existing self-play approaches for large language models often plateau during extended training due to the problem generator producing artificially complex yet ineffective questions. This work proposes a Self-Guided Self-Play (SGS) framework that introduces, for the first time, an endogenous guidance mechanism within the model itself. In SGS, the language model simultaneously assumes three roles—solver, problem generator, and guide—with the guide evaluating the relevance and naturalness of synthesized problems to effectively suppress reward hacking and problem degradation. Evaluated in the Lean4 formal theorem-proving environment, SGS surpasses the asymptotic solving rate of the strongest reinforcement learning baseline in fewer than 80 self-play rounds. After 200 training rounds, a 7B-parameter model trained with SGS solves more problems than the pass@4 result of a 671B-parameter model.

This work addresses the challenges of low training efficiency and suboptimal performance commonly faced by reinforcement learning agents in high-dimensional action spaces. It proposes, for the first time, a method that leverages large language models to dynamically generate action-level curricula, constructing multi-stage training trajectories for both Tabular Q-Learning and Deep Q-Network (DQN) agents in the game of Blackjack. By progressively introducing more complex actions, the approach integrates large language models, curriculum learning, and deep reinforcement learning to enhance learning efficacy. Evaluated in an eight-deck Blackjack environment, the method significantly improves agent performance: the DQN agent’s win rate increases from 43.97% to 47.41%, its bust rate decreases from 32.9% to 28.0%, and training converges over 74% faster—requiring less total training time than the evaluation phase of baseline methods.

This study addresses the challenge of automating the development of general-purpose capabilities in language models through curriculum learning. It proposes a "cognitive training" framework that treats language models as agents and leverages cross-entropy games to generate task curricula, iteratively expanding their skill boundaries via greedy optimization combined with a meta-sampling mechanism. The core contribution lies in the first formalization of the meta-objective for building general capabilities, whose uniqueness—under natural assumptions and involving only a few hyperparameters—is rigorously established, thereby providing a theoretical foundation for curriculum learning. This work identifies cognitive training as a viable pathway toward autonomous, continuous discovery and evolution of skills in language models.

This work addresses the challenges of sparse rewards, weak credit assignment, and scarce labeled data in deep-search agents for complex information retrieval by proposing π-Play, a multi-agent self-evolution framework. π-Play leverages question construction paths (QCPs)—naturally emerging during self-play—as privileged context to enable privileged self-distillation, thereby generating dense supervision signals. This mechanism transforms conventional sparse-reward self-play into an efficient self-evolution loop. Notably, without relying on human feedback or external annotations, π-Play surpasses fully supervised search agents under zero external data conditions and achieves a 2–3× improvement in evolution efficiency over traditional self-play methods.

This work addresses the decision-making challenges in Big 2, a four-player imperfect-information card game characterized by hidden information, sparse rewards, and non-stationary opponents. The authors propose a unified self-play reinforcement learning framework to systematically compare policy gradient and value approximation methods. Their findings indicate that moderate entropy regularization effectively mitigates excessive policy determinism, and that training against the current policy outperforms checkpoint-based or fixed-opponent mechanisms under limited training budgets. Empirical results demonstrate that Proximal Policy Optimization (PPO) with entropy regularization significantly outperforms alternative algorithms—including MC-Q, SARSA, and Q-learning—when facing random, greedy, and heuristic opponents. These results validate Big 2 as a compelling benchmark for multi-agent reinforcement learning in imperfect-information settings.