CineSubBench: Evaluating LLMs on Long-Form Narrative and Cultural Understanding from Multilingual Movie Subtitles

2026-09-28

AI Summary

This study addresses the absence of systematic evaluation for large language models in long-context narrative comprehension and cross-cultural understanding by pioneering film as a long-context evaluation domain. Leveraging 1,012 films and 8.13 million timestamped subtitles, we construct a unified multilingual benchmark dataset and design seven tasks to comprehensively assess models’ deep understanding of plot causality and cultural contexts. Our experiments reveal significant performance disparities across models in event reconstruction, cross-lingual consistency, and hierarchical calibration. Furthermore, we demonstrate that strong profanity is localized more accurately than mild vulgarity. By establishing film-based narratives as a rigorous testbed, this work introduces a novel paradigm for multimodal long-context evaluation.

Abstract

Large language models are increasingly evaluated in specialized domains such as law, medicine, software engineering, and cybersecurity, yet film remains comparatively underexplored despite requiring long-form narrative integration, multilingual interpretation, and culturally situated audience judgments. We introduce CineSubBench, a benchmark for evaluating long-context film understanding from multilingual movie subtitles. A subtitle track represents a film as thousands of short, temporally ordered utterances from which models must reconstruct characters, relationships, events, causal progression, and themes without explicit scene or event structure. CineSubBench contains 1,012 films with complete subtitle coverage in six languages, yielding 6,072 tracks and 8.13M timestamped subtitle entries. It provides a matched multi-task, multilingual, and multicultural (MultiX) evaluation setting: seven tasks span narrative reconstruction and abstraction, genre prediction, age suitability, country-specific motion-picture ratings across ten national classification systems, and subtitle-grounded language safety. Across nine LLMs, plot premises are recovered more reliably than event-complete synopses; cross-lingual consistency varies substantially across models and languages; national rating systems expose distinct calibration patterns; and strong profanity is far easier to ground than mild obscenity. CineSubBench establishes film as a long-context LLM evaluation domain and provides a unified benchmark for measuring narrative, multilingual, cultural, and evidence-grounding capabilities.

Problem

• Large Language Models
• Long-context Understanding
• Multilingual Evaluation
• Cultural Understanding
• Movie Subtitles

Innovation

• Long-context evaluation
• Multilingual movie subtitles
• Benchmark
• Narrative understanding
• Cross-cultural assessment