Learning to Optimize through Solver-Grounded Self-Play

2026-09-28

AI Summary

This study addresses the generalization bottlenecks and capability anchoring inherent in existing LLM optimization modeling, which relies heavily on external data. To overcome these limitations, this work proposes OPT-Zero, the first self-play training framework that operates entirely without external data. OPT-Zero employs a single LLM to simultaneously assume the dual roles of Proposer and Solver, achieving unsupervised automatic curriculum learning through closed-loop co-evolution. By integrating execution-feedback reinforcement learning with self-synthesis techniques, the framework completely eliminates dependence on human annotations or teacher models. Experimental results demonstrate that OPT-Zero matches state-of-the-art performance using zero external planning data while significantly enhancing model generalization capabilities, thereby establishing a new, scalable paradigm for LLM self-play training.

Abstract

Optimization modeling is central to many decision-making scenarios, but traditionally requires extensive domain expertise. While Large Language Models (LLMs) have shown promise in automating this process, current training paradigms mainly rely on human-annotated or teacher-generated datasets. This dependence introduces a Generalization Ceiling, where models overfit to narrow data distributions, and Capability Anchoring, where models'reasoning is bounded by annotator proficiency and teacher model capability. In response, we propose OPT-Zero, the first fully self-play training framework for optimization modeling that requires zero external training data. OPT-Zero employs a single LLM in a dual-role closed loop: a Proposer that synthesizes increasingly challenging optimization problems alongside their mathematical formulations and solving code, and a Solver that attempts to resolve the problems given only natural-language problem descriptions. Grounded in execution feedback from external optimization solvers, we alternately train both roles using reinforcement learning. This process fosters an auto-curriculum in which the Proposer and Solver co-evolve: generating harder valid problems by the Proposer seamlessly enhances the structural reasoning ability of the Solver. Extensive results indicate that with zero curated data, OPT-Zero matches state-of-the-art data-dependent methods while exhibiting substantially stronger generalizability, establishing self-play training as a highly scalable paradigm for advancing LLM reasoning in modeling and solving optimization problems.

Problem

• Optimization Modeling
• Large Language Models
• Self-Play
• Generalization Ceiling
• Capability Anchoring

Innovation

• Self-Play
• Optimization Modeling
• Reinforcement Learning
• Auto-Curriculum
• Zero Data