🤖 AI Summary
This study addresses the limitation of existing surrogate model evaluations, which rely on global metrics that overlook local consistency structures and fail to capture distortions near decision boundaries or teacher model boundary instability. We propose a boundary-aware conditional evaluation framework that defines confidence regions and integrates two boundary identification techniques, conducting systematic empirical analyses across multiple datasets and surrogate models. Our findings reveal that surrogate fidelity degrades significantly near teacher decision boundaries; moreover, even when global and boundary fidelities are comparable, different surrogates exhibit markedly divergent responses to perturbations. This work demonstrates the superiority of conditional evaluation over global scoring, emphasizing the need to account for where consistency is distributed and how stable teacher boundaries remain across training runs, thereby offering a new paradigm for enhancing predictive robustness.
📝 Abstract
Surrogate models are commonly evaluated by how often they agree with their teacher model over an evaluation set. Local variation in this agreement is well known, but its structure and consequences are less clear. We ask whether disagreement is systematically concentrated near the teacher's decision boundary and whether retaining that structure provides information beyond a single global score. Across several datasets and surrogate model classes, we find substantially lower fidelity near teacher decision boundaries under two different methods of identifying near-boundary examples. Moreover, conditioning agreement on confidence-defined regions improves prediction of teacher--surrogate agreement when evaluation-set composition changes, relative to the global score alone. Yet surrogates that agree equally well with the teacher both globally and near the decision boundary can respond very differently to changes selected using the surrogate itself. Finally, we show that independently trained deep teachers can agree on most predictions while identifying different examples as lying near their decision boundaries, complicating the use of those boundaries as stable reference regions for evaluating surrogates. Together, these results show that surrogate fidelity depends not only on how often a surrogate agrees with its teacher, but also on where that agreement holds and, for deep models, how stable the teacher's decision boundary is across training runs.