Mind the Gap: Disentangling Performance Bottlenecks in Video Instance Segmentation

2026-06-05

AI Summary

The performance bottlenecks among classification, segmentation, and tracking in video instance segmentation remain unclear, hindering targeted improvements. This work proposes the first systematic error decoupling framework, formulating identity and category assignment as an integer linear programming (ILP) problem to construct a model-agnostic diagnostic oracle, accompanied by a visualization tool named TrackLens. Analyses of seven methods on YouTube-VIS and OVIS reveal that tracking instability is the primary bottleneck, stemming from algorithmic temporal fragility rather than representational capacity. Online methods suffer a tracking AP drop exceeding 20% under severe occlusion, with performance degrading sharply as video length and instance density increase—an issue not substantially mitigated by stronger backbone networks.

Abstract

In Video Instance Segmentation (VIS), classification, segmentation, and tracking objectives are jointly evaluated, but their individual contributions to performance loss remain opaque. We introduce a diagnostic framework that formulates identity and class assignment as an Integer Linear Program (ILP), yielding a model-agnostic oracle that hierarchically isolates each error source. Applied to seven VIS methods spanning online and offline paradigms across YouTube-VIS 2019/2021 and a diagnostic subset of OVIS, our analysis reveals a consistent picture. Tracking instability is a critical bottleneck for online methods, with gaps exceeding 20 AP under heavy occlusion, and grows sharply with video length and instance density. While semantic classification contributes meaningfully on standard benchmarks, its impact becomes negligible where tracking fails most. Although stronger backbones substantially lift default scores, they leave AP tracking gaps largely intact, confirming that temporal fragility is algorithmic rather than purely representational. To complement the oracle, we introduce TrackLens, a visual tool that translates gap magnitude into observable, query-level failure modes. Together, these tools provide a systematic foundation for targeting VIS's core challenge: robust long-term temporal association.

Problem

• Video Instance Segmentation
• Performance Bottlenecks
• Tracking Instability
• Temporal Association
• Error Attribution

Innovation

• Video Instance Segmentation
• Performance Disentanglement
• Integer Linear Programming
• Tracking Bottleneck
• Model-agnostic Diagnosis