SkipClick: Combining Quick Responses and Low-Level Features for Interactive Segmentation in Winter Sports Contexts

2025-01-14

AI Summary

This work addresses interactive segmentation of skiing equipment and human bodies in winter sports imagery. We propose SkipClick—a lightweight encoder-decoder architecture incorporating cross-layer skip-guidance modules and click-aware low-level feature reweighting to achieve both rapid response and high-precision segmentation. Specifically optimized for click-based prompting, SkipClick is jointly trained on the WSESeg and HQSeg-44k benchmarks alongside a newly constructed skier mask dataset. Experiments demonstrate substantial improvements in annotation efficiency: on WSESeg, the number of clicks required to achieve 85% IoU (NoC@85) is reduced by 2.34 and 7.95 compared to SAM and HQ-SAM, respectively; on HQSeg-44k, NoC@90 = 6.00 and NoC@95 = 9.89—setting new state-of-the-art performance. The core contribution lies in the first joint modeling of dynamic low-level feature reweighting and skip-connected click guidance, effectively balancing real-time inference capability with fine-grained segmentation accuracy.

Technology Category

• Computer Vision: Segmentation
• Search and Optimization: Sampling/Simulation-based Search
• Humans and AI: Interaction Techniques and Devices

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

In this paper, we present a novel architecture for interactive segmentation in winter sports contexts. The field of interactive segmentation deals with the prediction of high-quality segmentation masks by informing the network about the objects position with the help of user guidance. In our case the guidance consists of click prompts. For this task, we first present a baseline architecture which is specifically geared towards quickly responding after each click. Afterwards, we motivate and describe a number of architectural modifications which improve the performance when tasked with segmenting winter sports equipment on the WSESeg dataset. With regards to the average NoC@85 metric on the WSESeg classes, we outperform SAM and HQ-SAM by 2.336 and 7.946 clicks, respectively. When applied to the HQSeg-44k dataset, our system delivers state-of-the-art results with a NoC@90 of 6.00 and NoC@95 of 9.89. In addition to that, we test our model on a novel dataset containing masks for humans during skiing.

Problem

• Winter Sports Scene
• Object Recognition and Segmentation
• Interactive Image Segmentation

Innovation

• SkipClick
• Interactive Object Segmentation
• Winter Sports Scenes