From Grasping to Speaking: Generative AI-Based Environment-Grounded VR Communication Training for Autistic Individuals

2026-07-23

AI Summary

This study addresses the challenges autistic individuals face in effectively integrating verbal communication with task-oriented environments in workplace settings. To this end, we propose a generative AI–driven virtual reality training system that systematically investigates, for the first time, the impact of embodied “Conversation + Object + Grasping” (C+O+G) interactions on communication training for autistic learners. The system integrates large language model agents, a VR environment, hand-grasp recognition, and context-aware dialogue generation to create an ecologically valid, workplace-relevant training paradigm. User studies demonstrate that the C+O+G modality significantly enhances task integration and user engagement, is consistently preferred by both autistic participants and vocational coaches, and exhibits comparable usability and cognitive load relative to alternative interaction modalities.

Abstract

Autistic individuals often face barriers in workplace communication, where soft skills are embedded within ongoing tasks and surrounding environment context, not in isolated verbal exchange. Recent work has introduced LLM-driven agents into VR-based communication training and proposed prompting schemas that let agents generate dialogue grounded in the VR environment and the user's hand-based interactions. Building on this work, we explore how different levels of environmental grounding influence the training experience of autistic trainees and job coaches. We conducted an exploratory study with 9 autistic trainees and 7 job coaches across three modalities: conversation-only (C), conversation with environmental objects (C+O), and conversation with objects and grasp interactions (C+O+G). Usability and workload were comparable across modalities, while both trainees and coaches preferred the more interactive and environment-grounded condition (C+O+G). Participants viewed C+O+G as helpful for sustaining engagement and integrating communication practice into task performance. We discuss design considerations for more flexible, interactive, and workplace-relevant VR communication training for autistic individuals.

Problem

• autism
• workplace communication
• virtual reality
• environmental grounding
• social skills training

Innovation

• Generative AI
• Environment-Grounded Interaction
• VR Communication Training
• Grasp-Based Interaction
• Autism Workplace Skills