TSCLIP: Robust CLIP Fine-Tuning for Worldwide Cross-Regional Traffic Sign Recognition

2024-09-23

arXiv.org

AI Summary

To address the significant degradation in model generalization caused by regional data distribution shifts in global cross-regional traffic sign recognition, this paper introduces the CLIP framework to this task for the first time and proposes a contrastive learning–based robust fine-tuning method. Key contributions include: (1) semantic-aware prompt engineering tailored to traffic sign characteristics; (2) an Adaptive Dynamic Weighted Ensemble (ADWE) mechanism that dynamically fuses features from multiple source domains; and (3) a novel multi-source, cross-regional benchmark dataset covering ten countries/regions. Experiments demonstrate that the proposed method substantially outperforms conventional deep learning models under mixed-domain evaluation, while preserving strong zero-shot transfer capability. This work establishes a new paradigm for traffic sign recognition in open-world scenarios—balancing robustness against domain shift and broad generalization across geographically diverse regions.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Adversarial Attacks & Robustness
• Application Domains: Transportation

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Security and Privacy: Large-scale security measurements

Abstract

Traffic sign is a critical map feature for navigation and traffic control. Nevertheless, current methods for traffic sign recognition rely on traditional deep learning models, which typically suffer from significant performance degradation considering the variations in data distribution across different regions. In this paper, we propose TSCLIP, a robust fine-tuning approach with the contrastive language-image pre-training (CLIP) model for worldwide cross-regional traffic sign recognition. We first curate a cross-regional traffic sign benchmark dataset by combining data from ten different sources. Then, we propose a prompt engineering scheme tailored to the characteristics of traffic signs, which involves specific scene descriptions and corresponding rules to generate targeted text descriptions. During the TSCLIP fine-tuning process, we implement adaptive dynamic weight ensembling (ADWE) to seamlessly incorporate outcomes from each training iteration with the zero-shot CLIP model. This approach ensures that the model retains its ability to generalize while acquiring new knowledge about traffic signs. To the best knowledge of authors, TSCLIP is the first contrastive language-image model used for the worldwide cross-regional traffic sign recognition task. The project website is available at: https://github.com/guoyangzhao/TSCLIP.

Problem

• Improves traffic sign recognition across regions
• Addresses data distribution variations in recognition
• Introduces CLIP model for global sign recognition

Innovation

• Uses CLIP model for traffic sign recognition
• Introduces adaptive dynamic weight ensembling
• Develops prompt engineering for traffic signs