Critical Design Strategy: a Method for Heuristically Evaluating Visualisation Designs

2025-08-07

AI Summary

To address novices’ insufficient reflective and critical evaluation skills in visualization design education, this paper proposes a structured, three-stage, six-perspective assessment framework. Grounded in six key dimensions—user, environment, interface, component, design process, and visual markup—the framework integrates 20 adjective-based rating scales and 30 heuristic questions to support systematic diagnostic analysis and iterative improvement. Its primary contribution lies in the first deep integration of critical thinking processes into visualization evaluation, yielding a teachable, reusable heuristic methodology. Validated over multiple years in undergraduate and graduate design courses, the framework demonstrably enhances learners’ design reflection capabilities. It has produced standardized pedagogical tools—including rubrics, workshop materials, and a practitioner’s guide—demonstrating strong educational applicability and practical utility for both teaching and visualization research and development.

Technology Category

• Data Mining & Knowledge Management: Data Visualization & Summarization
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Computer Vision: Visual Reasoning & Symbolic Representations

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Web Mining and Content Analysis: Web data visualization
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

We present the Critical Design Strategy (CDS) - a structured method designed to facilitate the examination of visualisation designs through reflection and critical thought. The CDS helps designers think critically and make informed improvements using heuristic evaluation. When developing a visual tool or pioneering a novel visualisation approach, identifying areas for enhancement can be challenging. Critical thinking is particularly crucial for visualisation designers and tool developers, especially those new to the field, such as studying visualisation in higher education. The CDS consists of three stages across six perspectives: Stage 1 captures the essence of the idea by assigning an indicative title and selecting five adjectives (from twenty options) to form initial impressions of the design. Stage 2 involves an in-depth critique using 30 heuristic questions spanning six key perspectives - user, environment, interface, components, design, and visual marks. Stage 3 focuses on synthesising insights, reflecting on design decisions, and determining the next steps forward. We introduce the CDS and explore its use across three visualisation modules in both undergraduate and postgraduate courses. Our longstanding experience with the CDS has allowed us to refine and develop it over time: from its initial creation through workshops in 2017/18 to improvements in wording and the development of two applications by 2020, followed by the expansion of support notes and refinement of heuristics through 2023; while using it in our teaching each year. This sustained use allows us to reflect on its practical application and offer guidance on how others can incorporate it into their own work.

Problem

• Facilitates critical evaluation of visualization designs
• Helps designers improve visual tools heuristically
• Supports novice visualization designers in education

Innovation

• Structured heuristic evaluation method for visualisations
• Three-stage process with six critical perspectives
• Refined through teaching and practical applications