Towards Intelligent Transportation with Pedestrians and Vehicles In-the-Loop: A Surveillance Video-Assisted Federated Digital Twin Framework

2025-03-06

AI Summary

Existing digital twin frameworks for intelligent transportation systems lack support for modeling strong pedestrian–vehicle interactions. Method: This paper proposes a video-driven federated digital twin architecture that enables an end-edge-cloud collaborative pedestrian–vehicle-in-the-loop simulation system. It introduces a novel semantic segmentation and twin-agent interaction modeling method based on multi-source surveillance videos, facilitating dynamic integration of localized digital twin systems (LDTS) across regions and enabling global real-time inference. Contribution/Results: The framework supports bidirectional, closed-loop pedestrian–vehicle-in-the-loop simulation. Experimental evaluation demonstrates a 42% reduction in mirroring latency, a 96.7% interaction recognition accuracy, and a user satisfaction score of 4.8/5.0—significantly outperforming conventional terminal-server architectures.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Computer Vision: Multi-modal Vision

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

In intelligent transportation systems (ITSs), incorporating pedestrians and vehicles in-the-loop is crucial for developing realistic and safe traffic management solutions. However, there is falls short of simulating complex real-world ITS scenarios, primarily due to the lack of a digital twin implementation framework for characterizing interactions between pedestrians and vehicles at different locations in different traffic environments. In this article, we propose a surveillance video assisted federated digital twin (SV-FDT) framework to empower ITSs with pedestrians and vehicles in-the-loop. Specifically, SVFDT builds comprehensive pedestrian-vehicle interaction models by leveraging multi-source traffic surveillance videos. Its architecture consists of three layers: (i) the end layer, which collects traffic surveillance videos from multiple sources; (ii) the edge layer, responsible for semantic segmentation-based visual understanding, twin agent-based interaction modeling, and local digital twin system (LDTS) creation in local regions; and (iii) the cloud layer, which integrates LDTSs across different regions to construct a global DT model in realtime. We analyze key design requirements and challenges and present core guidelines for SVFDT's system implementation. A testbed evaluation demonstrates its effectiveness in optimizing traffic management. Comparisons with traditional terminal-server frameworks highlight SV-FDT's advantages in mirroring delays, recognition accuracy, and subjective evaluation. Finally, we identify some open challenges and discuss future research directions.

Problem

• Lack of digital twin framework for pedestrian-vehicle interactions.
• Need for realistic traffic management in intelligent transportation systems.
• Challenges in simulating complex real-world traffic scenarios.

Innovation

• Surveillance video-assisted federated digital twin framework
• Multi-source traffic surveillance for interaction modeling
• Three-layer architecture for real-time global DT model