Complexity Scaling Laws for Neural Models using Combinatorial Optimization

2025-06-15

AI Summary

This work addresses neural combinatorial optimization, focusing on problem-intrinsic complexity—not computational resources—as the primary scaling variable. Method: Using the Traveling Salesman Problem (TSP) as a canonical benchmark, we quantitatively model two fundamental complexity measures: solution-space size and representation-space size. We combine combinatorial analysis with empirical studies across reinforcement learning, supervised fine-tuning, and local-search gradient descent, analyzing cost-landscape dynamics and generalization behavior. Contribution/Results: We discover that suboptimality scales as a power law with respect to node count or embedding dimension—robustly across training paradigms. Critically, gradient-based local search on the cost landscape reproduces identical scaling, confirming that the law arises from problem structure rather than algorithmic choices. This yields the first smooth, interpretable, problem-complexity-driven scaling theory for neural combinatorial optimization—providing a principled framework for predicting and understanding performance limits in learned solvers.

Technology Category

• Search and Optimization: Combinatorial Optimization
• Natural Language Processing: Learning & Optimization for NLP
• Machine Learning: Optimization

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Recent work on neural scaling laws demonstrates that model performance scales predictably with compute budget, model size, and dataset size. In this work, we develop scaling laws based on problem complexity. We analyze two fundamental complexity measures: solution space size and representation space size. Using the Traveling Salesman Problem (TSP) as a case study, we show that combinatorial optimization promotes smooth cost trends, and therefore meaningful scaling laws can be obtained even in the absence of an interpretable loss. We then show that suboptimality grows predictably for fixed-size models when scaling the number of TSP nodes or spatial dimensions, independent of whether the model was trained with reinforcement learning or supervised fine-tuning on a static dataset. We conclude with an analogy to problem complexity scaling in local search, showing that a much simpler gradient descent of the cost landscape produces similar trends.

Problem

• Develop scaling laws based on problem complexity measures
• Analyze solution and representation space size in neural models
• Study suboptimality growth in fixed-size models for TSP

Innovation

• Scaling laws based on problem complexity measures
• Combinatorial optimization ensures smooth cost trends
• Suboptimality grows predictably with scaling parameters