Surrogate assisted diversity estimation in neural ensemble search

2026-07-29

AI Summary

Neural ensemble search faces significant challenges due to the exponential growth of the joint search space over individual architectures and ensemble compositions, rendering brute-force approaches computationally infeasible. To address this, this work proposes a bi-objective surrogate modeling approach that introduces, for the first time, a co-optimization mechanism: two independently trained surrogate models estimate both the predictive accuracy and diversity potential of candidate architectures. Leveraging a directed acyclic graph representation of architectures, the method efficiently guides the ensemble search process. Experimental results on FashionMNIST, CIFAR-10, and CIFAR-100 demonstrate that the resulting ensembles achieve performance comparable to or better than established baselines such as Deep Ensembles and random search, while substantially reducing computational complexity.

Abstract

Ensembles are a standard way to improve the performance and robustness of deep neural networks, but their effectiveness crucially depends on both the quality and the diversity of individual models. Most neural architecture search (NAS) methods are computationally expensive. Extending them to neural ensemble search (NES), which requires joint optimization of individual architectures and their ensemble composition, leads to an exponential growth of the search space and makes the problem computationally intractable. To address this, we introduce a dual-objective surrogate-guided ensemble search: candidate architectures are represented as directed acyclic graphs, and two surrogate models are trained independently to estimate predictive accuracy and diversity potential. Their combined estimates guide an NES framework that efficiently identifies architectures that are both individually strong and collectively diverse. Our final ensemble achieves competitive or superior performance compared to standard baselines such as Deep Ensembles and Random Search on FashionMNIST, CIFAR-10, and CIFAR-100.

Problem

• neural ensemble search
• diversity estimation
• surrogate model
• neural architecture search
• computational tractability

Innovation

• surrogate-assisted search
• neural ensemble search
• diversity estimation
• dual-objective optimization
• neural architecture search