Chartist: Task-driven Eye Movement Control for Chart Reading

2025-02-05

AI Summary

This work addresses the challenge of task-dependent eye movement patterns in data visualization—where gaze behavior varies significantly across analytical tasks such as numeric retrieval, filtering, and extremum detection. We propose a task-driven eye-tracking trajectory prediction model built upon a two-level hierarchical control architecture: (1) a high-level large language model (LLM) that parses task semantics, infers the current analytical goal, and integrates previously acquired visual information; and (2) a low-level reinforcement learning–based sampling policy that dynamically generates fixation sequences aligned with human visual attention mechanisms. To our knowledge, this is the first approach to synergistically combine LLM-based task understanding with RL-driven oculomotor control for generating human-like, task-adaptive scanpaths. Experiments demonstrate high fidelity in simulating diverse analytical gaze behaviors. The model holds significant promise for visualization design evaluation, explainable AI, and optimization of interactive analytical systems.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

To design data visualizations that are easy to comprehend, we need to understand how people with different interests read them. Computational models of predicting scanpaths on charts could complement empirical studies by offering estimates of user performance inexpensively; however, previous models have been limited to gaze patterns and overlooked the effects of tasks. Here, we contribute Chartist, a computational model that simulates how users move their eyes to extract information from the chart in order to perform analysis tasks, including value retrieval, filtering, and finding extremes. The novel contribution lies in a two-level hierarchical control architecture. At the high level, the model uses LLMs to comprehend the information gained so far and applies this representation to select a goal for the lower-level controllers, which, in turn, move the eyes in accordance with a sampling policy learned via reinforcement learning. The model is capable of predicting human-like task-driven scanpaths across various tasks. It can be applied in fields such as explainable AI, visualization design evaluation, and optimization. While it displays limitations in terms of generalizability and accuracy, it takes modeling in a promising direction, toward understanding human behaviors in interacting with charts.

Problem

• Predicting task-driven eye movements
• Modeling user behavior in chart reading
• Enhancing data visualization comprehension

Innovation

• LLMs for goal selection
• Reinforcement learning for eye movement
• Hierarchical control architecture