Saudi Sign Language Translation Using T5

2025-10-13

AI Summary

This work addresses the unique challenges of Saudi Sign Language (SSL) translation, particularly face occlusion prevalent in regional cultural contexts. To this end, we construct the first SSL parallel corpus featuring realistic occlusion scenarios and propose three hierarchical evaluation protocols to comprehensively assess robustness. Methodologically, we explore cross-lingual transfer learning based on the T5 architecture: pretraining on YouTubeASL followed by fine-tuning on SSL data. Experiments demonstrate that this strategy improves BLEU-4 by approximately threefold over a baseline trained solely on SSL data. Our key contributions are: (1) releasing the first SSL translation dataset annotated for facial occlusion and grounded in Middle Eastern sociocultural context; (2) providing the first empirical validation of effective cross-sign-language transfer from American Sign Language (ASL) to SSL, revealing cross-lingual representational transferability in sign language models; and (3) introducing a novel evaluation paradigm for occlusion-robust sign language translation.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Security and Privacy: Large-scale security measurements
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

This paper explores the application of T5 models for Saudi Sign Language (SSL) translation using a novel dataset. The SSL dataset includes three challenging testing protocols, enabling comprehensive evaluation across different scenarios. Additionally, it captures unique SSL characteristics, such as face coverings, which pose challenges for sign recognition and translation. In our experiments, we investigate the impact of pre-training on American Sign Language (ASL) data by comparing T5 models pre-trained on the YouTubeASL dataset with models trained directly on the SSL dataset. Experimental results demonstrate that pre-training on YouTubeASL significantly improves models' performance (roughly $3 imes$ in BLEU-4), indicating cross-linguistic transferability in sign language models. Our findings highlight the benefits of leveraging large-scale ASL data to improve SSL translation and provide insights into the development of more effective sign language translation systems. Our code is publicly available at our GitHub repository.

Problem

• Translating Saudi Sign Language using T5 models with novel dataset
• Addressing unique SSL challenges like face coverings in recognition
• Investigating cross-linguistic transfer from ASL pre-training to SSL

Innovation

• T5 models applied to Saudi Sign Language translation
• Pre-training on YouTubeASL dataset improves SSL performance
• Novel SSL dataset includes face covering challenges