SoftPQ: Robust Instance Segmentation Evaluation via Soft Matching and Tunable Thresholds

2025-05-17

AI Summary

Traditional instance segmentation evaluation relies on binary matching at a fixed IoU threshold, limiting error-type discrimination and sensitivity to incremental improvements. To address this, we propose SoftPQ—a flexible generalization of Panoptic Quality—that replaces hard matching with a continuous, gradient-based instance association process. SoftPQ introduces tunable lower and upper IoU thresholds to define a partial-matching interval, and employs a sublinear penalty function to quantify ambiguity and fragmentation in predictions, enabling adjustable error sensitivity through soft matching. Empirically, SoftPQ significantly improves robustness to structural errors. Controlled perturbation experiments demonstrate its superior discriminative power and smoother, more informative score progression compared to standard PQ. By providing fine-grained, differentiable feedback, SoftPQ enhances benchmark evaluation and facilitates precise model optimization.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Segmentation
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Segmentation evaluation metrics traditionally rely on binary decision logic: predictions are either correct or incorrect, based on rigid IoU thresholds. Detection--based metrics such as F1 and mAP determine correctness at the object level using fixed overlap cutoffs, while overlap--based metrics like Intersection over Union (IoU) and Dice operate at the pixel level, often overlooking instance--level structure. Panoptic Quality (PQ) attempts to unify detection and segmentation assessment, but it remains dependent on hard-threshold matching--treating predictions below the threshold as entirely incorrect. This binary framing obscures important distinctions between qualitatively different errors and fails to reward gradual model improvements. We propose SoftPQ, a flexible and interpretable instance segmentation metric that redefines evaluation as a graded continuum rather than a binary classification. SoftPQ introduces tunable upper and lower IoU thresholds to define a partial matching region and applies a sublinear penalty function to ambiguous or fragmented predictions. These extensions allow SoftPQ to exhibit smoother score behavior, greater robustness to structural segmentation errors, and more informative feedback for model development and evaluation. Through controlled perturbation experiments, we show that SoftPQ captures meaningful differences in segmentation quality that existing metrics overlook, making it a practical and principled alternative for both benchmarking and iterative model refinement.

Problem

• Traditional metrics use rigid IoU thresholds for binary evaluation
• Existing methods overlook instance-level structure and gradual improvements
• SoftPQ introduces flexible thresholds for graded segmentation assessment

Innovation

• SoftPQ uses tunable IoU thresholds
• Introduces partial matching region
• Applies sublinear penalty function