Building and Evaluating a Realistic Virtual World for Large Scale Urban Exploration from 360° Videos

2025-10-13

AI Summary

To address high modeling costs, weak immersion, and poor interactivity in large-scale urban virtual tours, this paper proposes a lightweight, interactive virtual city construction method based on 360° street-view video. Eschewing conventional 3D reconstruction, our approach leverages video segmentation, intersection detection, semantic segmentation, and distortion-corrected spherical projection to enable building-level collision awareness and dynamic viewpoint rendering. Integrated with browser-side streaming, it supports real-time Web-based navigation. To the best of our knowledge, this is the first method to construct large-scale virtual urban environments with physics-aware interaction capabilities using only 360° video inputs. User studies demonstrate statistically significant improvements in presence (p < 0.01) and interaction naturalness over existing street-view interfaces. The proposed framework establishes a novel paradigm for low-cost, high-fidelity, city-scale virtual exploration.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Security and Privacy: Large-scale security measurements

Abstract

We propose to build realistic virtual worlds, called 360RVW, for large urban environments directly from 360° videos. We provide an interface for interactive exploration, where users can freely navigate via their own avatars. 360° videos record the entire environment of the shooting location simultaneously leading to highly realistic and immersive representations. Our system uses 360° videos recorded along streets and builds a 360RVW through four main operations: video segmentation by intersection detection, video completion to remove the videographer, semantic segmentation for virtual collision detection with the avatar, and projection onto a distorted sphere that moves along the camera trajectory following the avatar's movements. Our interface allows users to explore large urban environments by changing their walking direction at intersections or choosing a new location by clicking on a map. Even without a 3D model, the users can experience collision with buildings using metadata produced by semantic segmentation. Furthermore, we stream the 360° videos so users can directly access 360RVW via their web browser. We fully evaluate our system, including a perceptual experiment comparing our approach to previous exploratory interfaces. The results confirm the quality of our system, especially regarding the presence of users and the interactive exploration, making it most suitable for a virtual tour of urban environments.

Problem

• Creating realistic virtual urban worlds from 360° videos
• Enabling interactive navigation with avatar movement and collision detection
• Streaming browser-based exploration without requiring 3D models

Innovation

• Creating virtual worlds from 360° street videos
• Enabling interactive navigation via avatar and map
• Streaming videos for web browser access