VTBench: A Multimodal Framework for Time-Series Classification with Chart-Based Representations

2026-04-29

AI Summary

This work addresses the limitations of existing time series classification methods, which predominantly rely on raw numerical values while overlooking highly interpretable visual representations and lacking systematic evaluation of diverse chart types and fusion strategies. To bridge this gap, the authors propose VTBench, a novel framework that systematically investigates the efficacy of lightweight visualizations—including line, area, bar, and scatter plots—in time series classification. They introduce a modular multimodal fusion architecture capable of supporting single-chart, multi-chart, and full-modal integration. Extensive experiments across 31 UCR datasets demonstrate that chart-based models excel in few-shot scenarios, multi-chart fusion enhances classification accuracy, and multimodal fusion effectively improves or stabilizes performance when visual and raw features are non-redundant. The study further distills practical guidelines for chart selection and fusion strategy design.

Abstract

Time-series classification (TSC) has advanced significantly with deep learning, yet most models rely solely on raw numerical inputs, overlooking alternative representations. While texture-based encodings such as Gramian Angular Fields (GAF) and Recurrence Plots (RP) convert time series into 2D images, they often require heavy preprocessing and yield less intuitive representations. In contrast, chart-based visualizations offer more interpretable alternatives and show promise in specific domains; however, their effectiveness remains underexplored, with limited systematic evaluation across chart types, visual encoding choices, and datasets. In this work, we introduce VTBench, a systematic and extensible framework that re-examines TSC through multimodal fusion of raw sequences and chart-based visualizations. VTBench generates lightweight, human-interpretable plots -- line, area, bar, and scatter, providing complementary views of the same signal. We develop a modular architecture supporting multiple fusion strategies, including single-chart visual-numerical fusion, multi-chart visual fusion, and full multimodal fusion with raw inputs. Through experiments across 31 UCR datasets, we show that: (1) chart-only models are competitive in selected settings, particularly on smaller datasets; (2) combining multiple chart types can improve accuracy by capturing complementary visual cues; and (3) multimodal models improve or maintain performance when visual features provide non-redundant information, but may degrade accuracy when they introduce redundancy. We further distill practical guidelines for selecting chart types, fusion strategies, and configurations. VTBench establishes a unified foundation for interpretable and effective multimodal time-series classification.

Problem

• time-series classification
• chart-based representations
• multimodal fusion
• interpretability
• visual encoding

Innovation

• multimodal fusion
• chart-based representation
• time-series classification
• interpretable visualization
• VTBench