Annotations Are Not All You Need: A Cross-modal Knowledge Transfer Network for Unsupervised Temporal Sentence Grounding

2026-05-28

AI Summary

This work addresses the challenge of unsupervised temporal sentence localization, where paired video-query annotations and precise segment boundaries are unavailable. To overcome this limitation, the authors introduce a cross-task knowledge transfer mechanism that leverages inexpensive labels from image-noun and video-verb tasks to extract appearance and motion knowledge, respectively, and transfers them to complex real-world scenarios. The proposed method integrates entity-aware object-guided appearance modeling, event-aware motion representation learning, and a “copy-and-paste”–based motion representation refinement strategy to achieve accurate cross-modal alignment without paired supervision. Experiments on ActivityNet Captions and Charades-STA demonstrate significant performance gains over existing unsupervised approaches, with results approaching those of certain supervised models.

Abstract

This paper addresses the task of temporal sentence grounding (TSG). Although many respectable works have made decent achievements in this important topic, they severely rely on massive expensive video-query paired annotations, which require a tremendous amount of human effort to collect in real-world applications. To this end, in this paper, we target a more practical but challenging TSG setting: unsupervised temporal sentence grounding, where both paired video-query and segment boundary annotations are unavailable during the network training. Considering that some other cross-modal tasks provide many easily available yet cheap labels, we tend to collect and transfer their simple cross-modal alignment knowledge into our complex scenarios: 1) We first explore the entity-aware object-guided appearance knowledge from the paired Image-Noun task, and adapt them into each independent video frame; 2) Then, we extract the event-aware action representation from the paired Video-Verb task, and further refine the action representation into more practical but complicated real-world cases by a newly proposed copy-paste approach; 3) By modulating and transferring both appearance and action knowledge into our challenging unsupervised task, our model can directly utilize this general knowledge to correlate videos and queries, and accurately retrieve the relevant segment without training. Extensive experiments on two challenging datasets (ActivityNet Captions and Charades-STA) show our effectiveness, outperforming existing unsupervised methods and even competitively beating supervised works.

Problem

• temporal sentence grounding
• unsupervised learning
• cross-modal knowledge transfer
• video-language alignment
• annotation-free

Innovation

• unsupervised temporal sentence grounding
• cross-modal knowledge transfer
• appearance-action knowledge modulation
• copy-paste augmentation
• zero-shot video-text alignment