BasketLiDAR: The First LiDAR-Camera Multimodal Dataset for Professional Basketball MOT

2025-08-21

AI Summary

To address the accuracy and real-time performance bottlenecks in multi-object tracking (MOT) for basketball—caused by high-density occlusion and complex, rapid motion—this paper introduces the first synchronized LiDAR–multi-view camera multimodal basketball dataset and proposes a novel 3D spatially aware tracking framework. It pioneers the integration of LiDAR into sports MOT, enabling centimeter-accurate 3D pose annotation and cross-modal identity-consistent alignment. We design a lightweight, real-time pure-LiDAR tracking pipeline alongside a deeply fused LiDAR–camera tracking strategy. Evaluated on 4,445 frames from real matches, our method achieves real-time inference (≥30 FPS) and improves MOTA by 12.7% under severe occlusion, significantly outperforming state-of-the-art vision-only approaches. This work establishes a new paradigm for robust, real-time 3D MOT in highly dynamic sports scenarios.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Security and Privacy: Tracking, profiling, and countermeasures against them
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Real-time 3D trajectory player tracking in sports plays a crucial role in tactical analysis, performance evaluation, and enhancing spectator experience. Traditional systems rely on multi-camera setups, but are constrained by the inherently two-dimensional nature of video data and the need for complex 3D reconstruction processing, making real-time analysis challenging. Basketball, in particular, represents one of the most difficult scenarios in the MOT field, as ten players move rapidly and complexly within a confined court space, with frequent occlusions caused by intense physical contact. To address these challenges, this paper constructs BasketLiDAR, the first multimodal dataset in the sports MOT field that combines LiDAR point clouds with synchronized multi-view camera footage in a professional basketball environment, and proposes a novel MOT framework that simultaneously achieves improved tracking accuracy and reduced computational cost. The BasketLiDAR dataset contains a total of 4,445 frames and 3,105 player IDs, with fully synchronized IDs between three LiDAR sensors and three multi-view cameras. We recorded 5-on-5 and 3-on-3 game data from actual professional basketball players, providing complete 3D positional information and ID annotations for each player. Based on this dataset, we developed a novel MOT algorithm that leverages LiDAR's high-precision 3D spatial information. The proposed method consists of a real-time tracking pipeline using LiDAR alone and a multimodal tracking pipeline that fuses LiDAR and camera data. Experimental results demonstrate that our approach achieves real-time operation, which was difficult with conventional camera-only methods, while achieving superior tracking performance even under occlusion conditions. The dataset is available upon request at: https://sites.google.com/keio.jp/keio-csg/projects/basket-lidar

Problem

• Tracks 3D player trajectories in real-time basketball games
• Overcomes occlusion challenges in multi-player fast motion scenarios
• Reduces computational cost while improving tracking accuracy

Innovation

• Combines LiDAR point clouds with camera footage
• Uses real-time tracking pipeline with LiDAR data
• Fuses multimodal data for improved occlusion handling