From Hand-Crafted Metrics to Evolved Training-Free Performance Predictors for Neural Architecture Search via Genetic Programming

2025-05-16

AI Summary

Neural architecture search (NAS) suffers from poor generalizability and human bias in hand-crafted zero-cost (ZC) proxies. To address this, we propose the first fully automated ZC metric evolution framework based on symbolic regression and genetic programming. Our method treats existing manual ZC metrics as primitive building blocks and employs evolutionary algorithms to automatically synthesize novel, compact, and semantically interpretable ZC predictors. Evaluated across 13 heterogeneous tasks in NAS-Bench-Suite-Zero, our evolved metrics consistently outperform all hand-designed baselines—achieving an average 12.7% improvement in Spearman correlation—demonstrating strong cross-search-space and cross-task robustness. Moreover, the framework enables high-performance NAS on a single GPU in just 15 minutes, offering both computational efficiency and scalability. This work establishes a paradigm shift from manual design to automated, interpretable, and generalizable ZC proxy discovery.

Technology Category

• Search and Optimization: Evolutionary Computation
• Machine Learning: Deep Neural Architectures and Foundation Models
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Web Mining and Content Analysis: Models for Web evolution

Abstract

Estimating the network performance using zero-cost (ZC) metrics has proven both its efficiency and efficacy in Neural Architecture Search (NAS). However, a notable limitation of most ZC proxies is their inconsistency, as reflected by the substantial variation in their performance across different problems. Furthermore, the design of existing ZC metrics is manual, involving a time-consuming trial-and-error process that requires substantial domain expertise. These challenges raise two critical questions: (1) Can we automate the design of ZC metrics? and (2) Can we utilize the existing hand-crafted ZC metrics to synthesize a more generalizable one? In this study, we propose a framework based on Symbolic Regression via Genetic Programming to automate the design of ZC metrics. Our framework is not only highly extensible but also capable of quickly producing a ZC metric with a strong positive rank correlation to true network performance across diverse NAS search spaces and tasks. Extensive experiments on 13 problems from NAS-Bench-Suite-Zero demonstrate that our automatically generated proxies consistently outperform hand-crafted alternatives. Using our evolved proxy metric as the search objective in an evolutionary algorithm, we could identify network architectures with competitive performance within 15 minutes using a single consumer GPU.

Problem

• Automating design of zero-cost metrics for neural architecture search
• Improving consistency of performance predictors across diverse tasks
• Synthesizing generalizable metrics from existing hand-crafted proxies

Innovation

• Automates ZC metric design via Genetic Programming
• Synthesizes generalizable metrics from hand-crafted ones
• Achieves competitive NAS performance in 15 minutes