Absolute Zero: Reinforced Self-play Reasoning with Zero Data

2025-05-06

AI Summary

This work addresses the challenge of enhancing reasoning capabilities of large language models (LLMs) in truly zero-data settings. We propose the “Absolute Zero Point” paradigm: a fully self-driven reinforcement learning framework that requires no human annotations or external data, leveraging only self-generated tasks, code-execution-based self-verification rewards, and closed-loop curriculum evolution. Methodologically, it integrates verifiable-reward reinforcement learning (RLVR), programmatically generated tasks, and self-supervised policy optimization. Our key contributions are: (i) the first fully data-free, single-model autonomous reasoning evolution framework; (ii) elimination of reliance on human annotations; (iii) state-of-the-art performance on programming and mathematical reasoning benchmarks—significantly outperforming zero-shot baselines trained on tens of thousands of human-labeled examples; and (iv) broad compatibility across diverse model scales and architectures, demonstrating strong generalizability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Reinforcement learning with verifiable rewards (RLVR) has shown promise in enhancing the reasoning capabilities of large language models by learning directly from outcome-based rewards. Recent RLVR works that operate under the zero setting avoid supervision in labeling the reasoning process, but still depend on manually curated collections of questions and answers for training. The scarcity of high-quality, human-produced examples raises concerns about the long-term scalability of relying on human supervision, a challenge already evident in the domain of language model pretraining. Furthermore, in a hypothetical future where AI surpasses human intelligence, tasks provided by humans may offer limited learning potential for a superintelligent system. To address these concerns, we propose a new RLVR paradigm called Absolute Zero, in which a single model learns to propose tasks that maximize its own learning progress and improves reasoning by solving them, without relying on any external data. Under this paradigm, we introduce the Absolute Zero Reasoner (AZR), a system that self-evolves its training curriculum and reasoning ability by using a code executor to both validate proposed code reasoning tasks and verify answers, serving as an unified source of verifiable reward to guide open-ended yet grounded learning. Despite being trained entirely without external data, AZR achieves overall SOTA performance on coding and mathematical reasoning tasks, outperforming existing zero-setting models that rely on tens of thousands of in-domain human-curated examples. Furthermore, we demonstrate that AZR can be effectively applied across different model scales and is compatible with various model classes.

Problem

• Enhancing reasoning without human-curated data
• Self-evolving training curriculum for AI
• Achieving SOTA performance without external data

Innovation

• Self-evolving training curriculum without external data
• Code executor validates tasks and verifies answers
• Achieves SOTA performance without human examples