🤖 AI Summary
Multi-object tracking (MOT) suffers from hand-crafted association rules, limiting simultaneous gains in efficiency and robustness.
Method: We propose Track-Detection Link Prediction (TDLP), the first framework to introduce link prediction into the detection-to-tracking paradigm. TDLP performs end-to-end learning of inter-frame associations between trajectories and detections, enabling data-driven joint modeling of geometric and heterogeneous features (e.g., pose and appearance). It employs a lightweight graph neural network backbone that encodes bounding-box geometry and supports optional multimodal feature fusion, optimized via a joint objective combining contrastive learning and structured matching loss.
Results: TDLP achieves state-of-the-art performance on MOT17 and MOT20 benchmarks, outperforming both leading detection-based trackers and end-to-end methods. Ablation studies confirm that link prediction significantly surpasses conventional metric learning for heterogeneous feature integration. The code is publicly available.
📝 Abstract
Multi-object tracking aims to maintain object identities over time by associating detections across video frames. Two dominant paradigms exist in literature: tracking-by-detection methods, which are computationally efficient but rely on handcrafted association heuristics, and end-to-end approaches, which learn association from data at the cost of higher computational complexity. We propose Track-Detection Link Prediction (TDLP), a tracking-by-detection method that performs per-frame association via link prediction between tracks and detections, i.e., by predicting the correct continuation of each track at every frame. TDLP is architecturally designed primarily for geometric features such as bounding boxes, while optionally incorporating additional cues, including pose and appearance. Unlike heuristic-based methods, TDLP learns association directly from data without handcrafted rules, while remaining modular and computationally efficient compared to end-to-end trackers. Extensive experiments on multiple benchmarks demonstrate that TDLP consistently surpasses state-of-the-art performance across both tracking-by-detection and end-to-end methods. Finally, we provide a detailed analysis comparing link prediction with metric learning-based association and show that link prediction is more effective, particularly when handling heterogeneous features such as detection bounding boxes. Our code is available at href{https://github.com/Robotmurlock/TDLP}{https://github.com/Robotmurlock/TDLP}.