Baseline-Free Policy Optimization for Neural Combinatorial Optimization

2026-06-08

AI Summary

This work addresses the instability and frequent training collapse of traditional REINFORCE algorithms in neural combinatorial optimization, which stem from reliance on rollout baselines that degrade in quality on complex instances. For the first time, it introduces baseline-free policy optimization methods from large model alignment—such as GRPO and P3O—into this domain. Built upon the RL4CO framework, the approach replaces explicit baselines with intra-batch advantage normalization, thereby eliminating the structural fragility associated with baseline maintenance. Evaluated on routing problems including TSP and CVRP, the method substantially enhances training robustness: it successfully avoids collapse on TSP-100 while achieving solution quality within 2% of the strong POMO baseline, all without requiring any external or rollout baselines.

Abstract

Neural combinatorial optimization (NCO) trains autoregressive policies to solve routing problems. The standard training algorithm, REINFORCE with a rollout baseline, requires maintaining and periodically updating a frozen copy of the policy for variance reduction. This baseline introduces a structural vulnerability: on harder instances, a poor baseline produces noisy gradient estimates that can destabilize training. We evaluate Group Relative Policy Optimization (GRPO), an algorithm from large language model alignment that eliminates the baseline entirely by normalizing advantages within groups of sampled trajectories. In a controlled comparison of five RL algorithms on TSP and CVRP benchmarks within the RL4CO framework, we find that: (i) GRPO avoids the training collapse observed with REINFORCE on TSP-100, where performance degrades from cost 9.8 to 52.1 immediately after the warmup phase and does not recover under extended training; (ii) at matched gradient updates, GRPO achieves solution quality within 2% of POMO, a strong AM-based multi-start baseline, while requiring no external baseline; and (iii) P3O, a pairwise preference algorithm also from the alignment literature, is competitive on TSP but shows higher variability on CVRP. These results identify GRPO as a promising baseline-free alternative for NCO, particularly in settings where baseline-dependent training becomes fragile.

Problem

• neural combinatorial optimization
• baseline instability
• training collapse
• variance reduction
• routing problems

Innovation

• baseline-free
• Group Relative Policy Optimization
• neural combinatorial optimization
• REINFORCE
• variance reduction