search space engineering

Neural architecture search (NAS) suffers from high computational cost, manual design of macro-architectural configurations (e.g., depth and width), and unfairness in existing proxy-based performance estimation due to lack of architecture adaptivity. To address these issues, this paper proposes a globally navigable macro-micro joint search framework. We introduce the first macro-micro decoupled search paradigm, enabling fully automated co-optimization of depth and width via a hybrid search space. Furthermore, we design an architecture-aware dynamic training approximation mechanism that delivers low-overhead, differentiated performance prediction. Our method achieves state-of-the-art results on EMNIST and KMNIST, outperforms prior approaches on CIFAR-10, CIFAR-100, and Fashion-MNIST, accelerates search by 2–4× over the fastest global NAS methods, and successfully transfers to face recognition tasks.

This paper addresses the inefficiency and lack of scalability of manual hyperparameter tuning in large-scale machine learning. It systematically surveys hyperparameter optimization (HPO), unifying and classifying five mainstream paradigms: random/low-discrepancy search, bandit-based methods, Bayesian optimization, population-based (evolutionary) algorithms, and gradient-based differentiable optimization. The survey further extends to emerging settings—including online HPO, constrained HPO, and multi-objective HPO. Crucially, the work establishes novel theoretical connections between HPO and meta-learning as well as neural architecture search, yielding a comprehensive knowledge framework that articulates methodological principles, applicability boundaries, and inherent limitations. By clarifying the technical evolution and identifying key open challenges, this study provides a theoretically grounded yet practically actionable foundation for automated machine learning.

Neural architecture search (NAS) faces significant challenges in highly expressive search spaces—such as context-free grammar–based structural spaces—including prohibitive evaluation costs and poor cross-dataset generalization. To address this, we propose a transferable surrogate modeling framework that jointly leverages zero-cost proxies, neural graph features (GRAF), and fine-tuned large language models to enable accurate cross-dataset architectural performance prediction. Our work is the first to systematically demonstrate strong generalization of such surrogate models in cross-domain settings, supporting both architecture pre-screening and direct substitution of expensive training objectives. The method substantially reduces NAS computational overhead, discovers superior architectures on unseen datasets, and achieves high prediction accuracy with robust transferability—effectively balancing search efficiency and final model performance.

This work addresses the high computational cost of conventional neural architecture search (NAS) performance predictors, which often rely on expensive fine-tuning or intricate architecture representations. The authors propose Code-Oriented Language Model Embeddings (COLE), a method that directly uses raw PyTorch class definition code as input and leverages a frozen off-the-shelf language model to extract architecture embeddings. Coupled with a lightweight regression head, COLE constructs an efficient performance predictor without requiring NAS-specific fine-tuning. Evaluated on the NAS-Bench-201 benchmark, COLE achieves within 1% of the optimal architecture’s accuracy while reducing the evaluation budget by 34% compared to path-based encoding. Furthermore, experiments on CIFAR-100 demonstrate its strong generalization capability and superior search efficiency.

This work proposes a bilevel neural architecture search (NAS) framework grounded in auxiliary mathematical programming, formulating NAS as a bilevel optimization problem that jointly optimizes outer-level architecture parameters and inner-level network weights. By explicitly incorporating second-order derivative information of the training loss, the method enables synchronous updates of architecture and model parameters while ensuring local optimality of the inner-level solution. The approach systematically integrates bilevel optimization theory with mathematical programming techniques to enhance both search efficiency and final model performance. Experimental results demonstrate that the proposed framework significantly outperforms conventional sampling-based NAS methods in terms of both accuracy and computational efficiency.

This work addresses the limitations of existing neural architecture search (NAS) methods, which are either confined to narrow predefined search spaces or suffer from inefficiency and bias when leveraging large language models (LLMs) for open-ended exploration. To overcome these challenges, the authors propose a semi-automated NAS framework that constructs a prior-informed, open search space by structurally modeling architectural knowledge extracted from scientific literature. The framework integrates the FairNAD algorithm with multiple fairness-aware mutation mechanisms—including fair sampling, Pareto-aware selection, and LLM-driven iterative refinement—to enable efficient, diverse, and high-quality architecture discovery. Empirical evaluations demonstrate consistent improvements over state-of-the-art methods, achieving accuracy gains of 0.84%, 2.17%, and 2.35% on CIFAR-10, CIFAR-100, and ImageNet16-120, respectively.

Hardware-aware neural architecture search (HW-NAS) faces dual challenges: joint optimization of accuracy and latency, and low search efficiency. Conventional supernet-based methods incur prohibitive computational overhead, while existing LLM-driven approaches suffer from exploration bias, limiting coverage of diverse architectures across latency regimes. This paper proposes an efficient LLM-driven HW-NAS framework. Its core innovations are: (1) a complexity-aware hierarchical partitioning of the search space; and (2) a co-evolutionary mechanism for architecture generation and prompt engineering, integrating zero-cost predictors and knowledge-base-guided prompt refinement to mitigate LLMs’ intrinsic architectural biases. Evaluated on HW-NAS-Bench, our method achieves higher hypervolume (HV), lower inverted generational distance (IGD), up to 54% latency reduction, and compresses search time from days to minutes—significantly improving both efficiency and diversity in discovering cost-effective architectures across latency ranges.

通过分析万余次实验，发现LLM智能体主要通过架构搜索（而非超参调优）提升性能，并验证其能自主发现高效模型结构。

Deep learning model scaling has led to surging computational and energy demands, severely hindering deployment in resource-constrained environments. To address this, we propose a systematic approximation framework for efficient and scalable deep learning. Our method reformulates discrete constraints—such as pruning and quantization—as differentiable continuous relaxations, enabling fine-grained joint compression; introduces a lightweight neural architecture search algorithm based on parameter sharing and implicit recurrent structures; and enhances adaptive optimizers to accelerate hyperparameter convergence. The framework supports end-to-end joint training. Evaluated on image classification, language modeling, and generative tasks, it achieves an average 2.1× throughput improvement and 47% memory reduction during training and inference, while maintaining or exceeding baseline model accuracy. This enables effective low-resource deployment without compromising performance.

This work addresses the inefficiency of conventional black-box optimization in hierarchical heterogeneous search spaces where a large fraction of configurations are infeasible due to crash risks. To tackle this challenge, the authors propose a feasibility-first exploration strategy combined with a novel Thermal Budget Annealing (TBA) method that decomposes optimization into distinct exploration and exploitation phases. Robustness is further enhanced through trial timeouts, subspace blacklisting, and Tree-structured Parzen Estimators (TPE). The study introduces DeployBench, a new benchmark featuring hidden crash zones and non-uniform evaluation costs. Experimental results demonstrate that the proposed approach significantly improves the success rate of discovering valid model families under constraints and substantially reduces the overhead of invalid evaluations in both synthetic and real-world GPU deployment tasks.