One missing piece in Vision and Language: A Survey on Comics Understanding

2024-09-14

arXiv.org

AI Summary

Comic understanding faces core challenges including highly variable artistic styles, nonlinear reading orders, strong visual-textual coupling, and loosely structured narratives. Method: This paper introduces LoCU (Layer of Comics Understanding), the first hierarchical task framework that unifies multi-level objectives—from low-level perception to high-level narrative reasoning—while establishing a structured survey paradigm through systematic analysis of 30+ datasets and methods. Contribution/Results: LoCU enables systematic categorization and critical analysis of existing approaches spanning multimodal vision-language models, document understanding, visual question answering, object detection, and narrative modeling. It identifies key bottlenecks, such as cross-page temporal modeling and style-robust inference, and releases Awesome Comics Understanding—an open-source resource repository—to establish an extensible methodological foundation and collaborative platform for intelligent comic research.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Vision-language models have recently evolved into versatile systems capable of high performance across a range of tasks, such as document understanding, visual question answering, and grounding, often in zero-shot settings. Comics Understanding, a complex and multifaceted field, stands to greatly benefit from these advances. Comics, as a medium, combine rich visual and textual narratives, challenging AI models with tasks that span image classification, object detection, instance segmentation, and deeper narrative comprehension through sequential panels. However, the unique structure of comics -- characterized by creative variations in style, reading order, and non-linear storytelling -- presents a set of challenges distinct from those in other visual-language domains. In this survey, we present a comprehensive review of Comics Understanding from both dataset and task perspectives. Our contributions are fivefold: (1) We analyze the structure of the comics medium, detailing its distinctive compositional elements; (2) We survey the widely used datasets and tasks in comics research, emphasizing their role in advancing the field; (3) We introduce the Layer of Comics Understanding (LoCU) framework, a novel taxonomy that redefines vision-language tasks within comics and lays the foundation for future work; (4) We provide a detailed review and categorization of existing methods following the LoCU framework; (5) Finally, we highlight current research challenges and propose directions for future exploration, particularly in the context of vision-language models applied to comics. This survey is the first to propose a task-oriented framework for comics intelligence and aims to guide future research by addressing critical gaps in data availability and task definition. A project associated with this survey is available at https://github.com/emanuelevivoli/awesome-comics-understanding.

Problem

• Comic Understanding
• Art Style Variability
• Non-Traditional Narrative

Innovation

• LoCU Classification
• Visual Language Tasks
• Comic Understanding Framework