On the Normalization of Confusion Matrices: Methods and Geometric Interpretations

2025-09-05

AI Summary

Confusion matrices are often confounded by both inter-class similarity and data distribution bias, making it difficult to disentangle their independent contributions to misclassification. To address this, we propose a doubly stochastic normalization method that achieves joint row- and column-wise normalization via iterative proportional fitting. This is the first approach to establish an explicit geometric correspondence between the normalized confusion matrix and the model’s class representation space. Our method effectively decouples distributional bias from intrinsic class confusion, thereby accurately recovering the underlying class similarity structure. Experiments demonstrate that the proposed normalization significantly improves the accuracy and interpretability of error pattern diagnosis. It enables fine-grained analysis of classifier behavior—distinguishing systematic biases due to imbalanced sampling from fundamental ambiguities arising from semantic or feature-space proximity—and provides a principled tool for classifier evaluation, debugging, and optimization.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

The confusion matrix is a standard tool for evaluating classifiers by providing insights into class-level errors. In heterogeneous settings, its values are shaped by two main factors: class similarity -- how easily the model confuses two classes -- and distribution bias, arising from skewed distributions in the training and test sets. However, confusion matrix values reflect a mix of both factors, making it difficult to disentangle their individual contributions. To address this, we introduce bistochastic normalization using Iterative Proportional Fitting, a generalization of row and column normalization. Unlike standard normalizations, this method recovers the underlying structure of class similarity. By disentangling error sources, it enables more accurate diagnosis of model behavior and supports more targeted improvements. We also show a correspondence between confusion matrix normalizations and the model's internal class representations. Both standard and bistochastic normalizations can be interpreted geometrically in this space, offering a deeper understanding of what normalization reveals about a classifier.

Problem

• Disentangling class similarity and distribution bias in confusion matrices
• Introducing bistochastic normalization to recover underlying class similarity structure
• Providing geometric interpretations of normalization methods for classifiers

Innovation

• Bistochastic normalization via Iterative Proportional Fitting
• Disentangles class similarity from distribution bias
• Geometric interpretation of normalization in representation space