ELMI: Interactive and Intelligent Sign Language Translation of Lyrics for Song Signing

2024-09-15

arXiv.org

AI Summary

Current sign language song translation faces challenges including semantic distortion, syntactic misalignment, and difficulty synchronizing affective expression with musical rhythm, compounded by a lack of accessible, domain-specific tools. This paper introduces the first paradigm integrating real-time audio-visual synchronization editing with large language model (LLM)-guided multidimensional dialogue—covering semantics, facial expression, and prosody—for sign language song translation. The system supports gloss-based visual morpheme editing, time-aligned audio-video embedding, and Deaf-centered interaction design. An LLM-driven collaborative translation interface, augmented by user behavior feedback, ensures cultural sensitivity and practical usability. In a field study involving 13 professional sign language interpreters, the tool significantly enhanced translators’ confidence and autonomy: all participants successfully completed original song translations, and their feedback was constructive, motivating, and information-rich.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

d/Deaf and hearing song-signers have become prevalent across video-sharing platforms, but translating songs into sign language remains cumbersome and inaccessible. Our formative study revealed the challenges song-signers face, including semantic, syntactic, expressive, and rhythmic considerations in translations. We present ELMI, an accessible song-signing tool that assists in translating lyrics into sign language. ELMI enables users to edit glosses line-by-line, with real-time synced lyric and music video snippets. Users can also chat with a large language model-driven AI to discuss meaning, glossing, emoting, and timing. Through an exploratory study with 13 song-signers, we examined how ELMI facilitates their workflows and how song-signers leverage and receive an LLM-driven chat for translation. Participants successfully adopted ELMI to song-signing, with active discussions throughout. They also reported improved confidence and independence in their translations, finding ELMI encouraging, constructive, and informative. We discuss research and design implications for accessible and culturally sensitive song-signing translation tools.

Problem

• Addresses inaccessibility in song-signing translation
• Integrates AI for real-time lyric translation assistance
• Enhances confidence in d/Deaf song-signers' translations

Innovation

• Interactive sign language translation tool
• Real-time synced lyric editing
• AI-driven chat for translation support