Subtype Robustness Is Not Just Accuracy: Calibration Under Unseen Subtype Shift

2026-08-01

AI Summary

This work introduces calibration as a novel dimension for evaluating fine-grained subtype robustness, examining model reliability when encountering subtypes unseen during training yet belonging to known coarse-grained categories. Systematic evaluation across ImageNet, BREEDS, iNaturalist, and CIFAR-100 on five mainstream architectures reveals that models exhibit severe overconfidence: their confidence fails to decrease appropriately when accuracy drops due to exposure to novel subtypes. This miscalibration under semantic shift is markedly more pronounced than under common image corruptions. Existing recalibration techniques and out-of-distribution detection methods only partially mitigate the issue, underscoring the necessity of treating calibration as an independent and essential criterion in robustness assessment.

Abstract

Subtype robustness asks whether a model keeps the correct coarse prediction when test examples come from fine-grained subtypes absent from training but still inside a known coarse category. Prior work studies this almost entirely through accuracy. We ask whether the model also stays calibrated. We present the first systematic study of the question across ImageNet, BREEDS, iNaturalist and CIFAR-100 with five architectures. Calibration breaks down on unseen subtypes, where accuracy drops while confidence barely follows, leaving the model systematically overconfident exactly where it has become less accurate. At matched accuracy loss, generic image corruption causes a much larger drop in confidence, so the effect is not a general consequence of losing accuracy. The model reacts to visible degradation but not to in-taxonomy novelty. Recalibration tuned on seen subtypes narrows the gap but does not close it, and out-of-distribution scores flag the affected inputs only weakly. Subtype robustness should therefore be evaluated through calibration, not accuracy alone.

Problem

• subtype robustness
• calibration
• unseen subtype shift
• model confidence
• distribution shift

Innovation

• subtype robustness
• model calibration
• unseen subtypes
• distribution shift
• overconfidence