Popcorn: A Configurable Benchmark for Visual Evidence in Multimodal Movie Recommendation

2026-06-08

AI Summary

Existing movie recommendation benchmarks predominantly rely on trailers, thumbnails, or metadata, making it difficult to systematically evaluate the distinct contributions of different visual evidence sources in long-form video recommendation. This work proposes a configurable multimodal movie recommendation benchmark that, for the first time, enables a systematic comparison of full-length movies, trailers, and thumbnails as visual evidence, revealing their non-interchangeability. Built upon modern vision and vision-language models, the benchmark extracts multimodal features from these sources and integrates them with MovieLens data under a unified protocol that standardizes modality combinations, fusion strategies, data splits, and evaluation procedures. To enhance reproducibility and extensibility, metadata is augmented using large language models. Experiments demonstrate that thumbnail-based vision-language representations offer efficient and scalable item-side representations, and that both the choice of visual source and fusion strategy significantly impact recommendation accuracy, coverage, diversity, and calibration.

Abstract

Movies are long-form audiovisual works, yet recommender benchmarks often rely on trailers, thumbnails, or metadata. These sources differ in semantics and scalability: full movies preserve consumption-level evidence, trailers concentrate promotional highlights, and thumbnails provide sparse but catalog-scale visual signals. We present Popcorn, a configurable benchmark for visual evidence in multimodal movie recommendation, combining title-aligned full-movie/trailer embeddings with MovieLens-linked thumbnail features encoded by modern visual and vision-language models. Popcorn standardizes modality assembly, fusion, splitting, evaluation, and LLM-augmented metadata through a single configuration contract. Experiments show that thumbnail VLMs provide strong, scalable item-side evidence, while controlled trailer/full-movie comparisons show that visual evidence sources are not interchangeable: the choice of source and fusion strategy affects ranking accuracy, coverage, diversity, and calibration. The framework is available at https://github.com/RecSys-lab/Popcorn.

Problem

• multimodal movie recommendation
• visual evidence
• benchmark
• full-movie
• thumbnail

Innovation

• multimodal recommendation
• visual evidence benchmark
• configurable framework
• vision-language models
• movie recommendation