Vibe Visualizing: How Visualization Novices Try (and Fail) to Generate and Interpret Visualizations with Conversational AI

2026-06-07

AI Summary

This study investigates the cognitive biases, erroneous behaviors, and coping strategies of visualization novices when generating and interpreting visualizations through interactions with conversational AI systems such as ChatGPT. Drawing on user study data—including dialogue logs, interviews, and questionnaires—the authors develop the first systematic coding framework tailored to conversational AI–assisted visualization and validate its generalizability by replaying prompts on Gemini and Claude. The analysis reveals four key themes: deficiencies in output quality, recurrent model errors, patterns of user misuse, and critical factors influencing trust and verification. The findings uncover distinct failure modes across different large language models and culminate in a set of generalizable design principles for AI-augmented visualization systems.

Abstract

Conversational AI has enabled users to generate and interpret visualizations through natural language, significantly lowering the technical barrier to entry. The increased accessibility brings visualization novices into data visualization, but also exposes them to misinformation and misinterpretations. We are motivated to examine what issues can arise in interactions with current conversational AI, whether visualization novices can recognize such issues, and how they respond to them. To examine these questions, we conducted a user study on ChatGPT with 20 visualization novices, collecting their conversation logs, semi-structured interview transcripts, and Likert-scale questionnaire responses. Through thematic analysis, we developed a codebook that covers AI execution compliance, issues of AI-generated visualizations, patterns of AI responses, and prompting patterns of users. We summarized four themes, including the quality of outcomes, recurring errors from ChatGPT, misuse by users, factors that affect user trust, confidence, and verification behavior, and human-AI collaboration dynamics. To demonstrate the generalizability of our codebook and findings, we replayed the initial user prompts on Gemini and Claude and compared the outcomes, which revealed distinct failure modes for each model. Based on the results of all analyses, we derive a set of design recommendations for future AI-assisted visualization systems. We conclude with discussions on literacy gaps, diverse human-AI collaboration dynamics, and implications for agentic visualization.

Problem

• conversational AI
• visualization novices
• misinterpretation
• AI-generated visualizations
• human-AI collaboration

Innovation

• conversational AI
• visualization novices
• human-AI collaboration
• failure modes
• agentic visualization