Test Case Prioritization for DNNs via Neural Collapse Instability

2026-07-22

AI Summary

This work addresses the limitations of existing test case prioritization methods for deep neural networks, which rely solely on single-checkpoint confidence and are thus vulnerable to “confident errors,” hindering early fault detection under limited testing budgets. To overcome this, the authors propose the NCIP framework, which, for the first time, incorporates geometric characteristics from the neural collapse phase into test prioritization. NCIP selects a representative subset of checkpoints based on the equiangularity of weight vectors—measured by the standard deviation of pairwise cosine similarities—and quantifies sample risk through prediction variability across these checkpoints. By replacing conventional confidence with cross-checkpoint instability, NCIP consistently outperforms baseline methods across multiple datasets and models, achieving relative improvements of 1.5%–16.6% in RAUC-ALL and 4.9%–20.6% in RAUC-500, with superior average performance overall.

Abstract

With the widespread deployment of deep neural networks (DNNs) in safety-critical domains, reducing the cost of model validation under limited testing budgets has become increasingly important. Existing test case prioritization techniques often rely on single-checkpoint confidence signals derived from output probabilities. However, DNNs can be confidently wrong, and the confidence margin between the predicted and competing classes is frequently small, which weakens early fault discovery. To address this limitation, we propose a Neural-Collapse-Inspired Prioritization (NCIP) framework that replaces absolute confidence with cross-checkpoint prediction variability in the terminal training regime, where model geometry becomes highly structured. NCIP introduces two key components. First, it selects an NC-guided representative subset of training checkpoints using an equiangularity score of classifier weights, quantified as the standard deviation of pairwise cosine similarities among class weight vectors. Second, it prioritizes test inputs by their prediction variability across the selected checkpoints, surfacing boundary-adjacent and failure-prone samples that are unstable under checkpoint-induced decision boundary shifts. Extensive experiments across multiple datasets and architectures show that NCIP achieves strong performance in early fault discovery compared with competitive baselines, with 1.5 to 16.6 percent RAUC-ALL gains and 4.9 to 20.6 percent RAUC-500 gains under the same testing budget. NCIP further attains the best average performance across all dataset-model pairs.

Problem

• Test Case Prioritization
• Deep Neural Networks
• Fault Discovery
• Confidence Calibration
• Model Validation

Innovation

• Neural Collapse
• Test Case Prioritization
• Prediction Variability
• Checkpoint Selection
• Deep Neural Networks