🤖 AI Summary
This study addresses the challenge of predicting generalization performance of deep learning models under distribution shift (non-IID settings). It presents the first systematic evaluation of over 40 generalization measures that rely solely on training data and model parameters, assessing their robustness across diverse architectures, training strategies, and types of distribution shifts. Through extensive hyperparameter search (>10,000 configurations) and comprehensive correlation analyses, the work uniquely incorporates calibration-based and information criterion–based metrics into the non-IID generalization evaluation framework. The findings reveal that the majority of existing measures substantially degrade under distribution shift, with only a few demonstrating consistent reliability. These results provide empirical grounding and practical guidance for evaluating generalization in non-IID scenarios.
📝 Abstract
Generalization remains a central yet unresolved challenge in deep learning, particularly the ability to predict a model's performance beyond its training distribution using quantities available prior to test-time evaluation. Building on the large-scale study of Jiang et al. (2020). and concerns by Dziugaite et al. (2020). about instability across training configurations, we benchmark the robustness of generalization measures beyond IID regime. We train small-to-medium models over 10,000 hyperparameter configurations and evaluate more than 40 measures computable from the trained model and the available training data alone. We significantly broaden the experimental scope along multiple axes: (i) extending the evaluation beyond the standard IID setting to include benchmarking for robustness across diverse distribution shifts, (ii) evaluating multiple architectures and training recipes, and (iii) newly incorporating calibration- and information-criteria-based measures to assess their alignment with both IID and OOD generalization. We find that distribution shifts can substantially alter the predictive performance of many generalization measures, while a smaller subset remains comparatively stable across settings.