SoccerNet-FoulRet: Retrieving Semantically Similar Soccer Foul Videos

2026-10-07

AI Summary

This study addresses the challenges of inconsistent refereeing decisions in soccer and the difficulty of retrieving semantically similar foul precedents. To this end, it proposes the first semantic foul video retrieval task defined by refereeing decision logic rather than visual similarity. Methodologically, an open benchmark dataset is constructed by integrating multi-view data, zero-shot video embeddings, vision-language models, and supervised fine-tuning, upon which various model architectures are systematically evaluated. Experimental results demonstrate that even the best-performing zero-shot model achieves a hit rate below 5%, while fine-tuning yields only marginal improvements. These findings underscore the extreme difficulty of the proposed task and establish it as a highly challenging open problem for the sports artificial intelligence community.

Abstract

Refereeing decisions in professional soccer remain inconsistent because referees cannot easily compare a contentious foul against similar past cases. We cast this as a retrieval problem and introduce SoccerNet-FoulRet, the first benchmark for semantic foul retrieval. Given a query foul, the task is to retrieve past fouls judged to be relevant precedents, regardless of camera angle, teams, or appearance. This differs from prior video-to-video retrieval, which matches clips by visual similarity or a shared event. Here, relevance is defined by refereeing interpretation. We build the benchmark from the SoccerNet-MVFoul dataset and evaluate retrieval ability of zero-shot video and vision-language embedders together with a task-specific fine-tuned baseline on 693 human-verified queries and category-relevance labels. Semantic foul retrieval remains challenging. The strongest zero-shot model achieves under 5% HitRate@10 on human-verified precedents, while category-supervised fine-tuning improves category relevance but transfers only modestly to precedent retrieval. We release SoccerNet-FoulRet to establish semantic foul retrieval as an open problem: https://github.com/SoccerNet/sn-foulret.

Problem

• Semantic Foul Retrieval
• Video Retrieval
• Soccer Refereeing
• Benchmark

Innovation

• Semantic Foul Retrieval
• Video-to-Video Retrieval
• Benchmark
• Vision-Language Embedders
• Zero-shot Retrieval