SALI: Shot-Aware Late Interaction for Cross-Shot Relation Matching in Text-to-Video Retrieval using Film-Grammar Knowledge

2026-09-24

AI Summary

This study addresses the limitation of single embeddings in capturing cross-shot character interactions for text-to-video retrieval by proposing SALI. Built upon the CLIP4Clip architecture, the method introduces a shot-aware late interaction mechanism and innovatively incorporates a cinematic grammar penalty to achieve fine-grained alignment between query subjects, objects, and visual shots. Furthermore, greedy maximum or optimal transport is employed as the matching operator. Experimental results demonstrate that SALI significantly improves the R@1 metric on multi-shot relational queries while maintaining stable overall recall rates, effectively enhancing retrieval performance in complex interactive scenarios.

Abstract

Text-to-video retrieval usually represents a video clip by a single embedding. This embedding often loses important relations between people. E.g., an interaction "Anna confronts Mark" is regularly filmed as alternating shot and reverse shot of both (Fig. 1a). No single shot or averaged embedding over clip shots captures this relation. Thus, we propose SALI (Shot-Aware Late Interaction). It extracts the subject and object from a single-sentence query, and matches the query, its subject and object text embeddings against each visual shot embedding of a video clip. The matching operator is greedy max or optimal transport. A film-grammar penalty in fine-tuning adds a small, consistent shift. Built on CLIP4Clip-meanP, SALI keeps overall recall on par on Condensed Movies and ActivityNet while raising R@1 on multi-shot relation queries by 3 and 12 points, the most among all compared methods, and improves such queries on MSR-VTT at a cost of 1.4 R@1 overall.

Problem

• text-to-video retrieval
• cross-shot relation matching
• video representation
• multi-shot interaction

Innovation

• Shot-Aware Late Interaction
• Text-to-Video Retrieval
• Optimal Transport
• Film-Grammar Knowledge
• Cross-Shot Relation Matching