Using Sign Language Production as Data Augmentation to enhance Sign Language Translation

2025-06-11

AI Summary

Sign language data scarcity severely limits the performance of translation models. To address this, we propose the first data augmentation framework leveraging sign language generation, integrating skeleton modeling, gesture concatenation, and two photorealistic generation methods—SignGAN and SignSplat—in a synergistic manner. Our approach simultaneously enhances signer appearance diversity and skeletal motion fidelity while preserving semantic consistency. By enriching low-resource training sets, it effectively mitigates overfitting and substantially improves model generalization. On standard benchmarks, our method achieves up to a 19% improvement in sign language translation accuracy, with particularly notable gains in few-shot settings—demonstrating enhanced robustness and precision. This work establishes a scalable, generation-aware data augmentation paradigm for low-resource sign language understanding.

Technology Category

• Natural Language Processing: Generation
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Machine learning models fundamentally rely on large quantities of high-quality data. Collecting the necessary data for these models can be challenging due to cost, scarcity, and privacy restrictions. Signed languages are visual languages used by the deaf community and are considered low-resource languages. Sign language datasets are often orders of magnitude smaller than their spoken language counterparts. Sign Language Production is the task of generating sign language videos from spoken language sentences, while Sign Language Translation is the reverse translation task. Here, we propose leveraging recent advancements in Sign Language Production to augment existing sign language datasets and enhance the performance of Sign Language Translation models. For this, we utilize three techniques: a skeleton-based approach to production, sign stitching, and two photo-realistic generative models, SignGAN and SignSplat. We evaluate the effectiveness of these techniques in enhancing the performance of Sign Language Translation models by generating variation in the signer's appearance and the motion of the skeletal data. Our results demonstrate that the proposed methods can effectively augment existing datasets and enhance the performance of Sign Language Translation models by up to 19%, paving the way for more robust and accurate Sign Language Translation systems, even in resource-constrained environments.

Problem

• Enhancing Sign Language Translation with data augmentation
• Addressing low-resource challenges in sign language datasets
• Improving translation models using Sign Language Production techniques

Innovation

• Skeleton-based approach for sign production
• Sign stitching to combine sign segments
• Photo-realistic generative models like SignGAN