Bar-JEPA: Extracting Values from Bar Chart with Joint-Embedding Predictive Architecture

2026-08-06

AI Summary

This work addresses the challenging task of automatically recovering underlying numerical values from bar charts, which is hindered by the scarcity and high cost of annotated data. The study introduces, for the first time, a Joint Embedding Predictive Architecture (JEPA) to chart understanding, proposing a self-supervised learning framework that leverages JEPA pretraining to extract semantically rich latent features. These features are then fed into a lightweight decoder to regress tick labels and bar coordinates, thereby reconstructing the original numerical values. The approach requires no large-scale labeled datasets and significantly outperforms end-to-end supervised baselines on numerical extraction tasks, demonstrating both the effectiveness and generalization capability of the learned representations. Code and data are publicly released.

Abstract

Bar charts are commonly used in data visualization, and while they are easily understood by humans, it is non-trivial to extract the underlying data computationally. For a machine-learning-based approach, training chart de-rendering models usually requires labeled, real-world data. Labeling data is a time consuming task, which is why annotated data is scarce. Models can learn more efficiently when provided with features of high semantic quality, which a joint-embedding predictive architecture (JEPA) is designed to learn in a self-supervised manner. We present a per-bar, numerical value recovery pipeline for bar charts, where a JEPA encoder is used to produce semantically rich latent features. The decoder model consuming these features is simple and quick to train and outputs the coordinates of ticks and bars, which can be used to recover bar values. The effectiveness of self-supervised finetuning and quality of the extracted features is evident when comparing our model to end-to-end supervised baselines. Code, datasets and checkpoints are available on \href{https://github.com/dralois/Bar-JEPA}{GitHub}.

Problem

• bar chart
• data extraction
• self-supervised learning
• chart understanding
• numerical recovery

Innovation

• JEPA
• self-supervised learning
• bar chart understanding
• chart de-rendering
• latent representation