TSVC:Tripartite Learning with Semantic Variation Consistency for Robust Image-Text Retrieval

2025-01-19

AI Summary

To address pervasive label noise in image–text cross-modal retrieval—where existing methods suffer from insufficient robustness and performance saturation due to homogeneous co-teaching—this paper proposes a novel tripartite collaborative learning framework comprising heterogeneous Coordinator, Master, and Assistant models, each specializing in noise-aware semantic alignment. Our key contributions are: (1) the first tripartite heterogeneous collaboration mechanism, overcoming limitations of conventional homogeneous co-training; (2) a soft-label estimation method based on mutual information variation; and (3) a semantic variation consistency constraint loss coupled with a noise-aware collaborative training strategy. Evaluated on three mainstream benchmarks, our approach achieves state-of-the-art retrieval accuracy under high-noise conditions, while significantly improving training stability and convergence speed.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Cross-modal retrieval maps data under different modality via semantic relevance. Existing approaches implicitly assume that data pairs are well-aligned and ignore the widely existing annotation noise, i.e., noisy correspondence (NC). Consequently, it inevitably causes performance degradation. Despite attempts that employ the co-teaching paradigm with identical architectures to provide distinct data perspectives, the differences between these architectures are primarily stemmed from random initialization. Thus, the model becomes increasingly homogeneous along with the training process. Consequently, the additional information brought by this paradigm is severely limited. In order to resolve this problem, we introduce a Tripartite learning with Semantic Variation Consistency (TSVC) for robust image-text retrieval. We design a tripartite cooperative learning mechanism comprising a Coordinator, a Master, and an Assistant model. The Coordinator distributes data, and the Assistant model supports the Master model's noisy label prediction with diverse data. Moreover, we introduce a soft label estimation method based on mutual information variation, which quantifies the noise in new samples and assigns corresponding soft labels. We also present a new loss function to enhance robustness and optimize training effectiveness. Extensive experiments on three widely used datasets demonstrate that, even at increasing noise ratios, TSVC exhibits significant advantages in retrieval accuracy and maintains stable training performance.

Problem

• Cross-modal retrieval
• Annotation errors
• Co-teaching effectiveness degradation

Innovation

• Triplet Semantic Learning
• Inter-modal Retrieval
• Error-resilient Annotation