Learning from Synchronization: Self-Supervised Uncalibrated Multi-View Person Association in Challenging Scenes

2025-03-17

AI Summary

To address multi-view person association under fully unsupervised conditions—i.e., without annotations or camera calibration—this paper proposes a self-supervised approach grounded in temporal synchronization. It leverages inherent temporal alignment across video frames from different views to construct self-supervised signals for learning joint geometric-appearance embeddings. Key contributions include: (i) the first time-synchronization pretraining task for multi-view person association; (ii) reprojection consistency and edge-association linearity constraints that require no ground-truth labels; and (iii) an encoder-decoder framework integrating multi-view geometry modeling with contrastive learning. Evaluated on three challenging benchmarks—WILDTRACK, MVOR, and SOLDIERS—the method significantly outperforms existing supervised and unsupervised approaches, achieving state-of-the-art performance.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Multi-view person association is a fundamental step towards multi-view analysis of human activities. Although the person re-identification features have been proven effective, they become unreliable in challenging scenes where persons share similar appearances. Therefore, cross-view geometric constraints are required for a more robust association. However, most existing approaches are either fully-supervised using ground-truth identity labels or require calibrated camera parameters that are hard to obtain. In this work, we investigate the potential of learning from synchronization, and propose a self-supervised uncalibrated multi-view person association approach, Self-MVA, without using any annotations. Specifically, we propose a self-supervised learning framework, consisting of an encoder-decoder model and a self-supervised pretext task, cross-view image synchronization, which aims to distinguish whether two images from different views are captured at the same time. The model encodes each person's unified geometric and appearance features, and we train it by utilizing synchronization labels for supervision after applying Hungarian matching to bridge the gap between instance-wise and image-wise distances. To further reduce the solution space, we propose two types of self-supervised linear constraints: multi-view re-projection and pairwise edge association. Extensive experiments on three challenging public benchmark datasets (WILDTRACK, MVOR, and SOLDIERS) show that our approach achieves state-of-the-art results, surpassing existing unsupervised and fully-supervised approaches. Code is available at https://github.com/CAMMA-public/Self-MVA.

Problem

• Multi-view person association in challenging scenes with similar appearances.
• Self-supervised learning without annotations or calibrated camera parameters.
• Improving robustness using cross-view geometric and appearance features.

Innovation

• Self-supervised uncalibrated multi-view person association
• Encoder-decoder model with cross-view synchronization
• Self-supervised linear constraints for reduced solution space