CaRe-Ego: Contact-aware Relationship Modeling for Egocentric Interactive Hand-object Segmentation

2024-07-08

AI Summary

Existing egocentric hand-object segmentation methods rely solely on visual features or hand predictions as auxiliary cues, neglecting hand-object contact relationships and inter-object physical couplings—leading to limited segmentation accuracy and poor generalization. This paper proposes an interaction-aware segmentation framework for first-person vision. We design a Hand-Guided Object Feature Enhancer (HOFE) that explicitly injects contact priors via hand-guided attention, and introduce a Contact-Centric Object Decoupling Strategy (CODS) to disentangle hand-object contact regions from inter-object physical associations. Together, these components jointly model fine-grained interactive structures. Our method achieves state-of-the-art performance on both in-domain and cross-domain benchmarks, with significant improvements in segmentation accuracy and generalization capability. The source code is publicly available.

Technology Category

• Computer Vision: Segmentation
• Humans and AI: Interaction Techniques and Devices
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Security and Privacy: Data transparency and provenance

Abstract

Egocentric Interactive hand-object segmentation (EgoIHOS) requires the segmentation of hands and interacting objects in egocentric images, which is crucial for understanding human behavior in assistive systems. Previous methods typically recognize hands and interacting objects as distinct semantic categories based solely on visual features, or simply use hand predictions as auxiliary cues for object segmentation. Despite the promising progress achieved by these methods, they fail to adequately model the interactive relationships between hands and objects while ignoring the coupled physical relationships among object categories, ultimately constraining their segmentation performance. To make up for the shortcomings of existing methods, we propose a novel method called CaRe-Ego that achieves state-of-the-art performance by emphasizing the contact between hands and objects from two aspects. First, we introduce a Hand-guided Object Feature Enhancer (HOFE) to establish the hand-object interactive relationships to extract more contact-relevant and discriminative object features. Second, we design the Contact-centric Object Decoupling Strategy (CODS) to explicitly model and disentangle coupling relationships among object categories, thereby emphasizing contact-aware feature learning. Experiments on various in-domain and out-of-domain test sets show that Care-Ego significantly outperforms existing methods with robust generalization capability. Codes are publicly available at https://github.com/yuggiehk/CaRe-Ego/.

Problem

• Modeling hand-object interactive relationships for egocentric segmentation
• Disentangling coupling relationships among object categories
• Improving contact-aware feature learning for better segmentation performance

Innovation

• Hand-guided Object Feature Enhancer for interactive relationships
• Contact-centric Object Decoupling Strategy for disentangling relationships
• Emphasizes contact-aware feature learning for segmentation