VRSL:Exploring the Comprehensibility of 360-Degree Camera Feeds for Sign Language Communication in Virtual Reality

2026-02-26

AI Summary

This study investigates the intelligibility of American Sign Language (ASL) videos captured using body-worn 360-degree cameras in virtual reality (VR) to enhance communication accessibility for deaf and hard-of-hearing users. ASL recordings were systematically acquired from three body-mounted positions—head, shoulders, and chest—and evaluated within a VR environment for sign recognition accuracy and user experience. This work presents the first systematic validation of this approach, demonstrating that the shoulder-mounted perspective yields the highest performance, achieving an 85% recognition accuracy and an overall comprehension success rate of 83.3%. Participants exhibited a strong preference for sign language over text-based communication. The study further identifies the adverse impact of peripheral distortion on intelligibility and offers design recommendations to optimize camera placement and mitigate visual distortion, thereby advancing accessible VR communication for sign language users.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

This study explores integrating sign language into virtual reality (VR) by examining the comprehensibility and user experience of viewing American Sign Language (ASL) videos captured with body-mounted 360-degree cameras. Ten participants identified ASL signs from videos recorded at three body-mounted positions: head, shoulder, and chest. Results showed the shoulder-mounted camera achieved the highest accuracy (85%), though differences between positions were not statistically significant. Participants noted that peripheral distortion in 360-degree videos impacted clarity, highlighting areas for improvement. Despite challenges, the overall comprehension success rate of 83.3% demonstrates the potential of video-based ASL communication in VR. Feedback emphasized the need to refine camera angles, reduce distortion, and explore alternative mounting positions. Participants expressed a preference for signing over text-based communication in VR, highlighting the importance of developing this approach to enhance accessibility and collaboration for Deaf and Hard of Hearing (DHH) users in virtual environments.

Problem

• sign language
• virtual reality
• 360-degree video
• comprehensibility
• accessibility

Innovation

• 360-degree camera
• sign language communication
• virtual reality
• American Sign Language (ASL)
• accessibility