confusion matrix analysis

Confusion matrices are often confounded by both inter-class similarity and data distribution bias, making it difficult to disentangle their independent contributions to misclassification. To address this, we propose a doubly stochastic normalization method that achieves joint row- and column-wise normalization via iterative proportional fitting. This is the first approach to establish an explicit geometric correspondence between the normalized confusion matrix and the model’s class representation space. Our method effectively decouples distributional bias from intrinsic class confusion, thereby accurately recovering the underlying class similarity structure. Experiments demonstrate that the proposed normalization significantly improves the accuracy and interpretability of error pattern diagnosis. It enables fine-grained analysis of classifier behavior—distinguishing systematic biases due to imbalanced sampling from fundamental ambiguities arising from semantic or feature-space proximity—and provides a principled tool for classifier evaluation, debugging, and optimization.

Existing classification performance metrics suffer from inconsistent scales and high sensitivity to class imbalance, rendering cross-dataset evaluation incomparable and difficult to interpret. To address this, we propose the Outperformance Score (OS), a unified normalization framework that maps any confusion-matrix-based metric onto the [0,1] interval. OS is defined as the percentile rank of an observed performance value within a reference empirical distribution induced by the class imbalance ratio. This constitutes the first general-purpose standardization framework that endows diverse CMBCP metrics with consistent semantics and comparable scales. Crucially, OS eliminates reliance on fixed decision thresholds or parametric distributional assumptions, enabling robust evaluation under dynamically varying imbalance ratios. Extensive validation across real-world datasets from healthcare, finance, and natural language processing domains demonstrates that OS significantly enhances reliability and consistency in both inter-metric and cross-dataset comparisons, while supporting plug-and-play integration.

To address the delayed performance monitoring caused by the absence of ground-truth labels post-deployment, this paper proposes the first label-free, multi-metric online estimation framework for binary classification tasks, supporting accuracy, precision, recall, and F1-score—metrics derived from the confusion matrix. Methodologically, it is the first to systematically generalize label-free performance estimation to *any* metric definable via the confusion matrix. By calibrating model confidence scores to model predictive probability distributions and leveraging stochastic variable propagation with Monte Carlo inference, the approach enables full probabilistic modeling of confusion matrix elements, yielding theoretically grounded point estimates and statistically valid confidence intervals. Experiments across multiple benchmark datasets demonstrate that the proposed method significantly outperforms existing baselines, achieving lower estimation errors across all metrics while maintaining nominal coverage rates for its confidence intervals.

Machine learning models frequently suffer unexpected failures in real-world deployment, hindering practical adoption. Method: This paper introduces, for the first time, an orthogonal dichotomy framework distinguishing reliability from robustness, formally characterizing model failure mechanisms from first principles and systematically mapping them to engineering practices and real-world deployment scenarios. Our approach integrates probabilistic modeling, uncertainty quantification, adversarial robustness analysis, distributional shift detection, and system-level fault tree analysis—bridging theoretical insights with industrial-grade diagnostic tools and canonical failure case studies. Contribution/Results: We deliver an actionable failure attribution guide comprising rigorous theoretical foundations, an open-source toolchain, and cross-domain application exemplars. The framework significantly enhances model trustworthiness, debuggability, and deployment success rates.

To address the limitation of prior-dependent constraints undermining generalizability in hierarchical multi-label classification (HMC), this paper proposes EDR, a prior-free error-driven constraint discovery framework. EDR automatically identifies model misprediction patterns to induce interpretable, structured logical constraints; it further integrates a constraint-driven post-processing mechanism with a neuro-symbolic joint modeling architecture to jointly perform error detection, constraint recovery, and multi-level consistency verification. For the first time, EDR achieves fully automated, interpretable knowledge discovery and robust cross-domain constraint transfer without predefined constraints—enabling effective constraint learning even under label noise. Evaluated on multiple public benchmarks and a newly constructed military vehicle recognition dataset, EDR achieves an error detection F1-score exceeding 0.89 and constraint recovery accuracy above 92%, significantly improving both hierarchical consistency and overall classification performance.

This work addresses the challenge of attributing misclassifications and evaluating robustness in black-box classifiers by proposing an explainability-aware optimization framework. The approach integrates L₀ sparsity regularization (XA-L₀) with a tolerance-region confusion matrix (TOR-Confusion Matrix) to generate minimal input perturbations that induce target predictions while preserving sparsity and semantic interpretability. This unified framework simultaneously enables the generation of highly interpretable counterfactual examples and fine-grained quantification of model robustness. Empirical evaluations on both image and tabular datasets demonstrate the method’s effectiveness, significantly outperforming existing black-box analysis techniques in terms of interpretability and robustness assessment fidelity.

This work addresses the semantic gap between evaluation metrics and training data in large model pretraining, which hinders precise diagnosis and remediation of capability deficiencies. The authors propose “capability slices” as fundamental units aligning evaluation and data, establishing a bidirectional classification framework that links evaluation tasks with non-instructional training data through explicit mapping rules. This enables a closed-loop pipeline from evaluation failures to targeted data interventions. For the first time, the approach supports auditable and systematic reasoning that translates evaluation signals into data corrections, moving beyond intuition-driven tuning paradigms. Experiments demonstrate its efficacy in both directions: repairing specific training loss components restores BBH performance to 66.44, while targeted data sampling boosts AIME2025/2026 Pass@128 from 6.67/0.00 to 26.67.

Whether class imbalance correction improves the performance of clinical prediction models remains controversial. This study leverages data from the GUSTO-I clinical trial to systematically evaluate the impact of various correction strategies—including algorithm-level rebalancing, oversampling, and hybrid sampling—on model discrimination (AUC), calibration (calibration plots and MAPE), and predictive stability (Classification Instability Index, CII) across varying sample sizes. Using penalized logistic regression with 200 bootstrap replications, we find that all correction methods fail to enhance discriminative performance and instead introduce greater calibration bias, risk overestimation, and increased prediction instability. These results challenge the common practice of routinely applying class imbalance corrections in clinical modeling and, for the first time in large-scale simulations, reveal their potential harms.

Current machine learning evaluation practices predominantly rely on surface-level performance metrics, often neglecting the internal mechanisms of models. This work proposes trustworthy interpretability as a central evaluation paradigm and, for the first time, systematically demonstrates that it satisfies core criteria from the philosophy of science—namely falsifiability, reproducibility, and predictive power. By constructing an evaluation framework that integrates causal analysis with mechanistic probing, the study delineates three functional pathways through which interpretability enables the identification of behavioral origins, detection of latent flaws, and prediction of potential failure modes. This approach advances model assessment beyond performance-oriented benchmarks toward a deeper understanding of underlying mechanisms.

Real-world datasets often suffer from multiple issues, including label noise, feature corruption, and spurious correlations, yet existing methods struggle to simultaneously identify erroneous samples and their specific error types with high precision. This work proposes DeMix, a novel framework that, for the first time, leverages influence vectors to characterize how individual training samples affect predictions on a validation set. By formulating data debugging as a multi-label classification task and incorporating intervention-based learning to extract invariant diagnostic criteria for each error type, DeMix enables joint identification of both corrupted samples and their underlying error categories. Evaluated across 11 benchmark tasks, DeMix improves the F1 score for data debugging by 22.61% on average and boosts downstream model performance by 9.32% after data repair, substantially outperforming current state-of-the-art approaches.