STAR: Skeletal Token Alignment and Rearrangement for Interaction Recognition

2026-07-19

AI Summary

This work addresses the challenges of modeling interaction cues and mitigating visual information scarcity in skeleton-based recognition of human-object and human-human interactions. To this end, the authors propose a novel multimodal fusion framework that aligns skeleton sequences and RGB videos in a shared latent space, leveraging visual cues to enrich skeletal representations. The method introduces entity rearrangement (ER) and interaction spatiotemporal tokens (ISTs) to capture fine-grained dependencies, along with a focus-on-interaction (FoI) strategy to extract visually relevant regions. Notably, it establishes an efficient paradigm that enables multimodal training while relying solely on skeleton data during inference. Extensive experiments demonstrate that the proposed approach significantly outperforms state-of-the-art methods across multiple benchmarks, including Chico, HARPER, and NTU Mutual 11/26, confirming its superior performance and strong generalization capability.

Abstract

Understanding physical human-robot and human-human interactions is a challenging yet emerging topic in 3D vision. While most existing methods rely on skeleton sequences--effective in low-light and privacy-sensitive environment--they face two major challenges: 1) learning and effectively exploiting interaction cues from skeletal data, and 2) compensating for the lack of visual information absent in skeletons alone. To address these challenges, we propose skeletal token alignment and rearrangement (STAR) for human-robot and human-human interaction recognition. It learns interaction-specific skeleton features and enriches them using visual cues by aligning skeleton and RGB video representations in a shared latent space. Specifically, STAR consists of three key components. First, we design a skeleton encoder that captures fine-grained interdependencies using Entity Rearrangement (ER) and Interactive Spatiotemporal Tokens (ISTs). Second, we present Visual Interaction Encoding that introduces a Focus on Interactions (FoI) strategy to attend to spatiotemporal regions relevant to interactions in RGB videos. Finally, these representations are aligned via a contrastive learning objective, with a refinement head further refines predictions. During training, STAR leverages both skeleton and RGB video data to learn robust, discriminative interaction representations. At inference time, it operates on skeletons alone, retaining visual-informed benefits while preserving skeleton-only efficiency. Extensive experiments on Chico, HARPER, NTU Mutual 11 and 26 datasets consistently validate our approach by demonstrating superior performance over state-of-the-art methods. Our code is publicly available at https://github.com/Necolizer/STAR.

Problem

• skeleton-based interaction recognition
• visual information deficiency
• interaction cues
• human-robot interaction
• human-human interaction

Innovation

• Skeletal Token Alignment
• Entity Rearrangement
• Interactive Spatiotemporal Tokens
• Focus on Interactions
• Contrastive Learning