Multimodal Data Comprehension: Understanding How Visual-Textual Chains of Information Influence Data Interpretation

2026-07-26

AI Summary

This study investigates how the presentation order of visual and textual modalities influences data comprehension. By designing two information sequences—“visualization-first” and “text-first”—and employing a combination of crowdsourced experiments, think-aloud protocols, and iterative description tasks, the research systematically demonstrates for the first time that modality order critically shapes interpretive strategies: a visualization-first approach fosters hypothesis-driven, exploratory understanding, whereas a text-first sequence induces confirmatory interpretation under framing effects. The work introduces a novel experimental paradigm for studying multimodal comprehension and provides both empirical evidence and theoretical grounding for designing effective data-driven multimodal communication.

Abstract

Visualizations and text often work together to support effective data communication. Despite this common paradigm, we know little about how the interplay of these modalities affects people's data comprehension. We present a novel experimental paradigm to investigate multimodal data comprehension---the process of people comprehending information from multimodal visual and textual data---across both crowdsourced and think-aloud environments. Our methodology employs two sequential chains for presenting multimodal information---a visualization-first chain and a text-first chain---asking people to describe the data presented iteratively. By comparing how people's data comprehension changes across the chain, we can assess the information contribution of each modality and how they shape subsequent comprehension. We found that the visualization-first chain facilitates exploratory comprehension with hypothesis-driven discovery, whereas the text-first chain yields confirmatory comprehension akin to framing effects where visualizations serve to reinforce and confirm observations drawn from text. Our findings provide empirical insights into multimodal information integration, with implications for designing more effective data-driven communication.

Problem

• multimodal comprehension
• visual-textual interaction
• data interpretation
• information integration
• visualization and text

Innovation

• multimodal comprehension
• visualization-first chain
• text-first chain
• data communication
• information integration