StreetViewAI: Making Street View Accessible Using Context-Aware Multimodal AI

2025-08-11

AI Summary

Existing street-level imagery tools (e.g., Google Street View) rely exclusively on visual interfaces, rendering them inaccessible to blind users. To address this gap, we present the first accessible street-view system designed specifically for blind individuals. Our approach integrates 360° image understanding, context-aware multimodal AI, voice-based interaction, and accessibility-optimized navigation controls—enabling open exploration of over 22 billion global street-level images and remote path planning. The system features end-to-end integration of computer vision, automatic speech recognition/synthesis, and inclusive interaction design, achieving the first fully immersive, blind-accessible street-view experience. Evaluated with 11 blind participants, it significantly improves point-of-interest discovery and route previewing capabilities while enhancing remote environmental awareness. Furthermore, we derive a reusable set of accessibility guidelines for street-view systems—filling a critical gap in both research and practice for non-visual map services.

Technology Category

• Humans and AI: AI for Accessibility
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Security and Privacy: Data transparency and provenance

Abstract

Interactive streetscape mapping tools such as Google Street View (GSV) and Meta Mapillary enable users to virtually navigate and experience real-world environments via immersive 360° imagery but remain fundamentally inaccessible to blind users. We introduce StreetViewAI, the first-ever accessible street view tool, which combines context-aware, multimodal AI, accessible navigation controls, and conversational speech. With StreetViewAI, blind users can virtually examine destinations, engage in open-world exploration, or virtually tour any of the over 220 billion images and 100+ countries where GSV is deployed. We iteratively designed StreetViewAI with a mixed-visual ability team and performed an evaluation with eleven blind users. Our findings demonstrate the value of an accessible street view in supporting POI investigations and remote route planning. We close by enumerating key guidelines for future work.

Problem

• Making street view accessible for blind users
• Combining AI and speech for virtual navigation
• Enabling exploration of 220+ billion GSV images

Innovation

• Context-aware multimodal AI for accessibility
• Conversational speech navigation controls
• Supports POI investigations and route planning