Read, Critique, or Sketch? Investigating Alternative Visualization Literacy Assessment Modalities

2026-07-31

AI Summary

This study addresses the limitations of existing visualization literacy assessments, which predominantly rely on multiple-choice questions and struggle to measure higher-order competencies due to ceiling effects. To overcome this, the work introduces two web-based qualitative assessment methods—visualization critique and sketching tasks—implemented through online think-aloud protocols and data-driven drawing exercises, respectively. These approaches holistically capture users’ advanced literacy in interpreting, evaluating, and constructing visualizations. Leveraging interactive tools, controlled comparative experiments, and validity evidence from established scales such as CALVI and Mini-VLAT, the proposed methods significantly differentiate individuals across varying expertise levels—including researchers, students, and crowdworkers—thereby surpassing the constraints of traditional item formats and uncovering critical competency dimensions previously unmeasurable by existing instruments.

Abstract

Visualization literacy is a multifaceted construct encompassing skills and competencies, such as decoding data, constructing charts, and identifying design flaws. Yet, assessments of these competencies has been primarily constrained to multiple choice assessments that target lower-order skills, such as chart comprehension. As a result, they often exhibit ceiling effects (i.e., even modestly skilled individuals commonly score near the top of the scale), and do not provide enough information about an individual's higher-order skills (e.g., applying external knowledge, formulating critiques, and designing visualizations). To close these gaps, we develop and investigate two web-based qualitative assessments for testing the critique and design aspects of visualization literacy through online think-aloud critique and sketching of visualization designs based on data and a prompt. We compare performance on our assessments to two established visualization literacy assessments, CALVI and Mini-VLAT, by administering them to three groups that represent three experience levels: crowdworkers, students who have taken a relevant course, and researchers. We find that our critique and sketching assessments capture skills distinct from existing measures and that they differentiate between experienced individuals better than multiple choice-based alternatives. Although administering and grading qualitative assessments can be challenging, our findings suggest qualitative, multimodal assessments are a promising complement to existing visualization literacy assessments, in particular when high visualization skills need to be distinguished.

Problem

• visualization literacy
• assessment
• higher-order skills
• ceiling effect
• qualitative evaluation

Innovation

• visualization literacy
• qualitative assessment
• critique
• sketching
• higher-order skills