Pluto: Authoring Semantically Aligned Text and Charts for Data-Driven Communication

2025-02-11

AI Summary

Existing visualization tools struggle to achieve semantic-level coordination between text and charts, hindering high-quality data storytelling. This paper introduces a hybrid active chart–text co-creation system featuring a novel bidirectional guidance mechanism that jointly models chart construction features and textual drafts. On one side, it parses visual encodings and aligns them with textual semantics to generate real-time suggestions—including annotations, title summarization, text completion, and data transformation recommendations. On the other, it supports brush-based interactions to trigger text generation and enables text-driven reverse inference for visualization refinement. The system integrates rule-based heuristic reasoning, visual encoding analysis, semantic similarity matching, and lightweight text generation—without relying on large language models—to ensure interpretability and author control. A user study demonstrates that our approach significantly reduces initial authoring time and achieves high recommendation adoption rates, empirically validating that semantic alignment enhances data storytelling quality.

Technology Category

• Data Mining & Knowledge Management: Data Visualization & Summarization
• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Web data visualization

Abstract

Textual content (including titles, annotations, and captions) plays a central role in helping readers understand a visualization by emphasizing, contextualizing, or summarizing the depicted data. Yet, existing visualization tools provide limited support for jointly authoring the two modalities of text and visuals such that both convey semantically-rich information and are cohesively integrated. In response, we introduce Pluto, a mixed-initiative authoring system that uses features of a chart's construction (e.g., visual encodings) as well as any textual descriptions a user may have drafted to make suggestions about the content and presentation of the two modalities. For instance, a user can begin to type out a description and interactively brush a region of interest in the chart, and Pluto will generate a relevant auto-completion of the sentence. Similarly, based on a written description, Pluto may suggest lifting a sentence out as an annotation or the visualization's title, or may suggest applying a data transformation (e.g., sort) to better align the two modalities. A preliminary user study revealed that Pluto's recommendations were particularly useful for bootstrapping the authoring process and helped identify different strategies participants adopt when jointly authoring text and charts. Based on study feedback, we discuss design implications for integrating interactive verification features between charts and text, offering control over text verbosity and tone, and enhancing the bidirectional flow in unified text and chart authoring tools.

Problem

• Enhances text-chart semantic alignment
• Supports joint text-visual authoring
• Facilitates interactive text-chart integration

Innovation

• Mixed-initiative authoring system
• Semantically integrated text and charts
• Interactive auto-completion and suggestions