GamerAstra: Enhancing Video Game Accessibility for Blind and Low-Vision Players through a Multi-Agent AI Framework

2025-06-28

AI Summary

Visually impaired and low-vision gamers face core accessibility barriers—including inaccessible visual game content, difficult UI navigation, and constrained interactions—while existing assistive solutions often require source-code modifications and lack cross-game generalizability. Method: We propose a source-code-agnostic multi-agent AI framework integrating large language models (LLMs) and vision-language models (VLMs), supporting multimodal inputs (e.g., speech, gestures) and enabling end-to-end real-time perception and decision-making via intelligent UI parsing. Contribution/Results: The framework introduces a novel customizable assistance granularity mechanism to accommodate diverse impairment severities. Evaluation demonstrates significant improvements in success rates for complex in-game actions and enhanced immersion across mainstream game genres. To our knowledge, this is the first approach achieving cross-game universal accessibility enhancement without any code-level intervention.

Technology Category

• Humans and AI: AI for Accessibility
• Computer Vision: Multi-modal Vision
• Multiagent Systems: Adversarial Agents

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Blind and low-vision (BLV) players encounter critical challenges in engaging with video games due to the inaccessibility of visual elements, difficulties in navigating interfaces, and limitations in sending interaction input. Moreover, the development of specialized accessibility features typically requires substantial programming effort and is often implemented on a game-by-game basis. To address these challenges, we introduce extit{GamerAstra}, a generalized accessibility framework that leverages a multi-agent design to facilitate access to video games for BLV players. It integrates multi-modal techniques including large language models and vision-language models, enabling interaction with games lacking native accessibility support. The framework further incorporates customizable assistance granularities to support varying degrees of visual impairment and enhances interface navigation through multiple input modalities. The evaluation through technical assessments and user studies indicate that extit{GamerAstra} effectively enhances playability and delivers a more immersive gaming experience for BLV players. These findings also underscore potential avenues for advancing intelligent accessibility frameworks in the gaming domain.

Problem

• Enhancing video game accessibility for blind and low-vision players
• Overcoming inaccessibility of visual elements and interface navigation
• Reducing programming effort for specialized accessibility features

Innovation

• Multi-agent AI framework for game accessibility
• Multi-modal LLM and vision-language integration
• Customizable assistance for varying visual impairments