Attention-Steered Vision-Language Models for Sign Language Translation

2026-07-31

AI Summary

Current vision-language models struggle to accurately localize critical spatiotemporal regions in sign language translation, limiting their performance. To address this issue, this work proposes AttnSign, a novel framework that introduces spatial attention supervision to explicitly guide the model toward key regions such as the face and hands. Additionally, it incorporates a reinforcement learning–based action rhythm guidance mechanism to modulate temporal attention over sign-level keyframes. Evaluated on the How2Sign and OpenASL benchmarks, the proposed method significantly outperforms existing approaches, demonstrating substantial improvements in both accuracy and robustness of sign language translation.

Abstract

Vision-language models (VLMs) have emerged as a powerful framework for multimodal video understanding. However, they remain limited in the sign language translation task, where we identify a key failure mode of existing VLMbased translators: poor spatial-temporal visual grounding. In particular, we find that standard next-token cross-entropy does not directly provide signal for where and when the model should attend, causing models to overlook sign-relevant regions and frames. To address this challenge, we propose AttnSign, a VLM-based spatial-temporal attention steering framework for sign language translation. AttnSign first introduces spatial attention supervision for sign-relevant regions, such as face and hands, in each frame; then develops an RL-based motion-cadence steering method that encourages the model to explore and focus on sign-level keyframes. Experimental results on How2Sign and OpenASL benchmarks show that our proposed AttnSign consistently outperforms existing methods.

Problem

• sign language translation
• vision-language models
• spatial-temporal grounding
• visual attention

Innovation

• attention steering
• vision-language models
• sign language translation
• spatial-temporal grounding
• reinforcement learning