FunL2O: LLM-Guided Feature Function Design for Learning to Optimize

2026-07-29

AI Summary

This work addresses the limitations of traditional Learning to Optimize (L2O) approaches, which rely on handcrafted feature functions that lack generalizability and automated representation learning. The authors propose FunL2O, a novel framework that integrates large language models (LLMs) with program evolution through a FunSearch-inspired iterative loop to automatically design executable feature functions. Within a fixed evaluation pipeline, the framework iteratively generates, validates, and refines features by combining LLMs, graph neural networks, and interfaces to diverse solvers. Experimental results demonstrate that across linear programming, quadratic programming, and mixed-integer optimization tasks, four distinct LLM-generated features consistently outperform manually designed counterparts, highlighting the method’s versatility, automation capability, and performance advantages in both continuous and discrete optimization domains.

Abstract

Learning-to-optimize (L2O) methods accelerate repeated optimization by training models to predict solutions, warm starts, branching decisions, or other forms of solver guidance. A critical yet largely overlooked component of these pipelines is the feature function that maps problem instances to inputs for machine learning models. Existing L2O methods typically rely on hand-crafted features, making representation design manual and largely fixed across domains. We introduce FunL2O, the first unified framework for automating feature design through LLM-driven program evolution for L2O. In a FunSearch-style loop, an LLM proposes executable feature functions, while a fixed evaluation process retrains the original L2O model and measures downstream optimization performance. We evaluate FunL2O on linear and quadratic programming tasks involving solution prediction and warm-starting, as well as on mixed-integer optimization tasks using GNN-guided backdoor branching and Predict-and-Search. Across continuous and discrete optimization tasks and four LLMs, the evolved features consistently outperform hand-crafted representations. These results establish LLM-driven feature evolution as a general and effective approach to automating representation design in L2O.

Problem

• Learning-to-Optimize
• feature function
• representation design
• automated feature engineering
• optimization

Innovation

• Learning-to-Optimize
• Feature Function Design
• Large Language Models
• Program Evolution
• Automated Representation Learning