Cross-Skeleton Interaction Graph Aggregation Network for Representation Learning of Mouse Social Behaviour

2022-08-07

arXiv.org

AI Summary

Existing mouse social behavior analysis methods struggle to model cross-skeleton keypoint interactions among multiple mice and lack robustness to individual morphological variations. To address this, we propose a social behavior modeling framework for multi-mouse skeletal sequences. Our method introduces a Cross-Skeleton Node-Level Interaction (CS-NLI) module and an interaction-aware Transformer to jointly model intra-skeleton, inter-skeleton, and cross-skeleton dynamic interactions. Furthermore, we design a node-similarity-based self-supervised task to enhance interaction representation learning. By integrating graph neural networks with self-supervised learning, our approach generates fine-grained, generalizable social behavior graph representations. Evaluated on CRMI13-Skeleton and our newly constructed PDMB-Skeleton datasets, the method significantly outperforms state-of-the-art approaches, achieving superior behavioral discriminability and cross-scenario generalization.

Abstract

Automated social behaviour analysis of mice has become an increasingly popular research area in behavioural neuroscience. Recently, pose information (i.e., locations of keypoints or skeleton) has been used to interpret social behaviours of mice. Nevertheless, effective encoding and decoding of social interaction information underlying the keypoints of mice has been rarely investigated in the existing methods. In particular, it is challenging to model complex social interactions between mice due to highly deformable body shapes and ambiguous movement patterns. To deal with the interaction modelling problem, we here propose a Cross-Skeleton Interaction Graph Aggregation Network (CS-IGANet) to learn abundant dynamics of freely interacting mice, where a Cross-Skeleton Node-level Interaction module (CS-NLI) is used to model multi-level interactions (i.e., intra-, inter- and cross-skeleton interactions). Furthermore, we design a novel Interaction-Aware Transformer (IAT) to dynamically learn the graph-level representation of social behaviours and update the node-level representation, guided by our proposed interaction-aware self-attention mechanism. Finally, to enhance the representation ability of our model, an auxiliary self-supervised learning task is proposed for measuring the similarity between cross-skeleton nodes. Experimental results on the standard CRMI13-Skeleton and our PDMB-Skeleton datasets show that our proposed model outperforms several other state-of-the-art approaches.

Problem

• Mouse Social Behavior
• Complex Interaction Analysis
• Body Morphology Variation

Innovation

• CS-IGANet
• IAT mechanism
• multi-layer interaction