🤖 AI Summary
This study investigates the impact of model selection criteria—such as accuracy versus loss—on test performance in neural classifier training, particularly under early stopping with patience. Through systematic empirical evaluation using k-fold cross-validation on standard benchmarks, the work compares multiple validation metrics, including accuracy, cross-entropy, C-Loss, and PolyLoss, under both early stopping and post-hoc full-trajectory selection strategies. The findings reveal that validation loss–based criteria consistently outperform validation accuracy, which exhibits not only inferior performance but also lower stability. More critically, regardless of the selection criterion employed, the chosen models are typically substantially worse than the best test performance observed during training, thereby exposing a fundamental limitation in current model selection paradigms.
📝 Abstract
Despite the extensive literature on training loss functions, the evaluation of generalization on the validation set remains underexplored. In this work, we conduct a systematic empirical and statistical study of how the validation criterion used for model selection affects test performance in neural classifiers, with attention to early stopping. Using fully connected networks on standard benchmarks under $k$-fold evaluation, we compare: (i) early stopping with patience and (ii) post-hoc selection over all epochs (i.e. no early stopping). Models are trained with cross-entropy, C-Loss, or PolyLoss; the model parameter selection on the validation set is made using accuracy or one of the three loss functions, each considered independently. Three main findings emerge. (1) Early stopping based on validation accuracy performs worst, consistently selecting checkpoints with lower test accuracy than both loss-based early stopping and post-hoc selection. (2) Loss-based validation criteria yield comparable and more stable test accuracy. (3) Across datasets and folds, any single validation rule often underperforms the test-optimal checkpoint. Overall, the selected model typically achieves test-set performance statistically lower than the best performance across all epochs, regardless of the validation criterion. Our results suggest avoiding validation accuracy (in particular with early stopping) for parameter selection, favoring loss-based validation criteria.