SiamNAS: Siamese Surrogate Model for Dominance Relation Prediction in Multi-objective Neural Architecture Search

2025-06-03

AI Summary

Multi-objective neural architecture search (NAS) suffers from high computational overhead in Pareto dominance evaluation and expensive crowding distance computation. This paper proposes a lightweight Siamese-network-based surrogate model that innovatively predicts pairwise architectural dominance directly—bypassing costly ground-truth evaluations and conventional crowding distance calculations. It further incorporates a model-size heuristic to efficiently maintain population diversity and supports multi-task co-optimization as well as Sets of Pareto Sets (SOS) generation. Evaluated on NAS-Bench-201, the method completes search in just 0.01 GPU-days, yielding the optimal architecture on CIFAR-10 and a near-optimal one on ImageNet-16-120. Dominance prediction accuracy reaches 92%, substantially reducing computational cost while significantly improving the efficiency of multi-objective NAS.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Deep Neural Architectures and Foundation Models
• Constraint Satisfaction and Optimization: Search

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Modern neural architecture search (NAS) is inherently multi-objective, balancing trade-offs such as accuracy, parameter count, and computational cost. This complexity makes NAS computationally expensive and nearly impossible to solve without efficient approximations. To address this, we propose a novel surrogate modelling approach that leverages an ensemble of Siamese network blocks to predict dominance relationships between candidate architectures. Lightweight and easy to train, the surrogate achieves 92% accuracy and replaces the crowding distance calculation in the survivor selection strategy with a heuristic rule based on model size. Integrated into a framework termed SiamNAS, this design eliminates costly evaluations during the search process. Experiments on NAS-Bench-201 demonstrate the framework's ability to identify Pareto-optimal solutions with significantly reduced computational costs. The proposed SiamNAS identified a final non-dominated set containing the best architecture in NAS-Bench-201 for CIFAR-10 and the second-best for ImageNet, in terms of test error rate, within 0.01 GPU days. This proof-of-concept study highlights the potential of the proposed Siamese network surrogate model to generalise to multi-tasking optimisation, enabling simultaneous optimisation across tasks. Additionally, it offers opportunities to extend the approach for generating Sets of Pareto Sets (SOS), providing diverse Pareto-optimal solutions for heterogeneous task settings.

Problem

• Predicts dominance relations in multi-objective NAS efficiently
• Reduces computational costs in neural architecture search
• Identifies Pareto-optimal solutions with high accuracy

Innovation

• Siamese network surrogate predicts dominance relationships
• Heuristic rule replaces crowding distance calculation
• SiamNAS reduces computational costs significantly