STARK: Spatio-Temporal Attention for Representation of Keypoints for Continuous Sign Language Recognition

2026-03-17

AI Summary

This work addresses the high parameter count and computational complexity of existing keypoint-based methods for continuous sign language recognition, which hinder efficient spatiotemporal modeling. The authors propose a unified spatiotemporal attention network that jointly captures spatial relationships among keypoints and their temporal dynamics within local time windows, yielding context-aware spatiotemporal representations. Notably, this approach introduces, for the first time, a unified spatiotemporal attention mechanism into keypoint modeling, replacing conventional combinations of graph convolution and 1D convolution, thereby substantially reducing model complexity. Evaluated on the Phoenix-14T dataset, the proposed method achieves performance comparable to state-of-the-art keypoint-based approaches while using 70–80% fewer parameters.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Data Mining & Knowledge Management: Mining of Spatial, Temporal or Spatio-Temporal Data
• Computer Vision: Language and Vision

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

Continuous Sign Language Recognition (CSLR) is a crucial task for understanding the languages of deaf communities. Contemporary keypoint-based approaches typically rely on spatio-temporal encoding, where spatial interactions among keypoints are modeled using Graph Convolutional Networks or attention mechanisms, while temporal dynamics are captured using 1D convolutional networks. However, such designs often introduce a large number of parameters in both the encoder and the decoder. This paper introduces a unified spatio-temporal attention network that computes attention scores both spatially (across keypoints) and temporally (within local windows), and aggregates features to produce a local context-aware spatio-temporal representation. The proposed encoder contains approximately $70-80\%$ fewer parameters than existing state-of-the-art models while achieving comparable performance to keypoint-based methods on the Phoenix-14T dataset.

Problem

• Continuous Sign Language Recognition
• Keypoints
• Spatio-Temporal Modeling
• Model Complexity
• Parameter Efficiency

Innovation

• spatio-temporal attention
• keypoint representation
• parameter efficiency
• continuous sign language recognition
• local context-aware