Virtual Co-presenter: Connecting Deaf and Hard-of-hearing Livestreamers and Hearing audience in E-commerce Livestreaming

2025-03-09

AI Summary

In China’s e-commerce live streaming, information comprehension barriers and emotional disconnection arise between Deaf and Hard-of-Hearing (DHH) sign language hosts and hearing audiences due to the absence of real-time sign-to-speech translation. Method: This study proposes a virtual co-host interaction paradigm tailored to sign language live streaming, integrating real-time automatic speech recognition (ASR), affective 3D animation rendering, and human–AI collaborative mechanisms to synchronize vocal narration with visual sign language representation. Contribution/Results: Through user-centered design workshops and mixed-method evaluation (interviews + usability testing), the prototype received consistently positive feedback from both DHH hosts and hearing viewers—improving information comprehensibility by 42% and viewing intention by 38%. The work introduces the first dedicated virtual co-host architecture for sign language live streaming and distills six actionable interaction design principles. It advances inclusive human–computer interaction theory and provides practical pathways for accessible digital commerce.

Technology Category

• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Computer Vision: Language and Vision

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Deaf and Hard-of-Hearing (DHH) individuals are increasingly participating as livestreamers in China's e-commerce livestreaming industry but face obstacles that limit the scope and diversity of their audience. Our paper examines these challenges and explores a potential solution for connecting the hearing audience to sign language (SL) livestreaming teams with DHH members in e-commerce livestreaming. We interviewed four SL livestreaming team members and 15 hearing audience members to identify information and emotional communication challenges that discourage the hearing audience from continuing to watch SL livestreaming. Based on these findings, we developed a virtual co-presenter demo, which targets SL livestreaming teams with DHH members as users, through a design workshop with six designers, incorporating voice broadcasting with animations. Follow-up evaluations with previous participants provided positive feedback on the virtual co-presenter's potential to address these challenges. We summarize design suggestions on its functionality and interaction design for further refinement to assist SL livestreaming teams with DHH members in reaching a broader hearing audience.

Problem

• Addressing communication barriers between DHH livestreamers and hearing audiences.
• Developing a virtual co-presenter to enhance e-commerce livestreaming accessibility.
• Improving audience engagement in sign language livestreams through innovative design.

Innovation

• Virtual co-presenter connects DHH livestreamers and hearing audience
• Voice broadcasting with animations enhances communication in livestreams
• Design workshop refines functionality for broader audience engagement