Compact set-valued deep ensembling in multi-class classification

2026-10-04

AI Summary

This study addresses the challenges of high training and storage overheads in deep ensemble learning, alongside its limited robustness in multi-utility prediction. To this end, it proposes a multi-class classification framework that integrates compact ensembles with set-valued prediction. Specifically, the method employs compact ensembles to generate probabilistic predictions and define representative distributions, subsequently computing optimal predictions for arbitrary utility functions based on Bayesian decision theory. Furthermore, it designs a family of efficiently solvable set-valued utilities and incorporates Bayesian neural networks, Monte Carlo dropout, and statistical distance optimization to facilitate model training. By effectively reconciling computational efficiency with predictive robustness, this work demonstrates empirically that the proposed framework significantly reduces resource consumption while enhancing prediction stability across diverse utility settings.

Abstract

This paper tackles visible challenges in deep ensemble learning, where deep neural networks serve as ensemble members: training and storage burdens, and robustness of cautious (set-valued) predictions targeting multiple utilities, which may involve reward-sensitivity. To mitigate the training and storage burdens, we propose to employ compact ensembles, such as Bayesian Neural Networks and Convolutional Neural Networks with the Monte-Carlo dropout prediction option, to produce probabilistic predictions. For each query instance, these probabilistic predictions are then used to define a representative distribution optimizing some statistical distance. The representative distribution is then employed to define the Bayes-optimal prediction (BOP) of any utility. To address the potential unrobustness of singleton prediction making, we propose a family of set-utilities satisfying some desirable properties and whose set-valued BOPs can be found efficiently. Empirical evidence is then given to illustrate the potential (dis)advantages of the proposed ensemble learning framework.

Problem

• deep ensemble learning
• set-valued prediction
• multi-class classification
• training and storage burden
• robustness

Innovation

• Deep Ensemble Learning
• Compact Ensembles
• Bayes-Optimal Prediction
• Set-valued Prediction
• Set-Utilities