🤖 AI Summary
This work proposes a bilevel neural architecture search (NAS) framework grounded in auxiliary mathematical programming, formulating NAS as a bilevel optimization problem that jointly optimizes outer-level architecture parameters and inner-level network weights. By explicitly incorporating second-order derivative information of the training loss, the method enables synchronous updates of architecture and model parameters while ensuring local optimality of the inner-level solution. The approach systematically integrates bilevel optimization theory with mathematical programming techniques to enhance both search efficiency and final model performance. Experimental results demonstrate that the proposed framework significantly outperforms conventional sampling-based NAS methods in terms of both accuracy and computational efficiency.
📝 Abstract
Bilevel optimization has become an influential and widely adopted framework for addressing hierarchical optimization problems in machine learning, providing an effective approach to modeling the interaction between two levels of optimization, with applications such as hyperparameter tuning, meta-learning, adversarial training, and data poisoning. Neural Architecture Search (NAS), a subfield of hyperparameter optimization, is a prime example of a bilevel optimization problem, with architecture parameters optimized at the outer-level and network weights optimized at the inner level. This paper presents a structured overview of NAS through the lens of bilevel optimization. We categorize existing NAS approaches into two main classes: sampling-based methods, which search optimal architectures using different architecture samplers, and bilevel theory-based methods, which solve the architecture search problem using bilevel optimization principles. We further highlight our current research direction, wherein the bilevel NAS formulation is addressed through an auxiliary mathematical programming framework. This framework enables the systematic integration of second-order information from the model's training loss function and ensures the optimality of the model parameters while modifying architecture parameters. By simultaneously updating the architecture and model parameters along their respective optimal descent directions derived from the auxiliary mathematical program, these methods achieve more principled and theoretically consistent results. The same auxiliary program can also be used for simultaneous hyperparameter and model fine-tuning. A comparative analysis shows that bilevel theory-based approaches generally outperform sampling-based methods, both in accuracy and efficiency.