Automated Algorithm Design for Auto-Tuning Optimizers

2025-10-19

AI Summary

Existing auto-tuning approaches suffer from poor optimizer generalizability and high manual design costs in high-dimensional, irregular parameter spaces. Method: This paper proposes a large language model (LLM)-driven paradigm for adaptive optimization algorithm generation. It jointly encodes problem descriptions and structured search-space representations, employs prompt engineering to guide the LLM in dynamically synthesizing customized search strategies, and refines these strategies via an iterative validation framework. Contribution/Results: Evaluated on four real-world high-performance computing scenarios, the generated optimizers outperform state-of-the-art methods by 72.4% on average. Ablation studies quantify the individual contributions of problem description modeling (30.7%) and search-space modeling (14.6%) to overall performance gain. To our knowledge, this is the first work to leverage LLMs for end-to-end optimization algorithm synthesis, overcoming the adaptability limitations inherent in fixed, hand-crafted optimizers.

Technology Category

Search and Optimization: Sampling/Simulation-based SearchMachine Learning: OptimizationNatural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationEconomics, Online Markets and Human Computation: Cost models of using LLMs in production systemsSearch and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Automatic performance tuning (auto-tuning) is essential for optimizing high-performance applications, where vast and irregular parameter spaces make manual exploration infeasible. Traditionally, auto-tuning relies on well-established optimization algorithms such as evolutionary algorithms, annealing methods, or surrogate model-based optimizers to efficiently find near-optimal configurations. However, designing effective optimizers remains challenging, as no single method performs best across all tuning tasks. In this work, we explore a new paradigm: using large language models (LLMs) to automatically generate optimization algorithms tailored to auto-tuning problems. We introduce a framework that prompts LLMs with problem descriptions and search-space characteristics results to produce specialized optimization strategies, which are iteratively examined and improved. These generated algorithms are evaluated on four real-world auto-tuning applications across six hardware platforms and compared against the state-of-the-art in optimization algorithms of two contemporary auto-tuning frameworks. The evaluation demonstrates that providing additional application- and search space-specific information in the generation stage results in an average performance improvement of 30.7% and 14.6%, respectively. In addition, our results show that LLM-generated optimizers can rival, and in various cases outperform, existing human-designed algorithms, with our best-performing generated optimization algorithms achieving, on average, 72.4% improvement over state-of-the-art optimizers for auto-tuning.

Problem

• Automating optimizer design using large language models
• Generating specialized algorithms for auto-tuning problems
• Improving performance over human-designed optimization methods

Innovation

• LLMs generate tailored optimization algorithms automatically
• Framework prompts LLMs with problem-specific search space details
• Generated algorithms outperform human-designed optimizers by 72.4%