ChartJudgeBench: Evaluating LMM Judges for Chart-to-Code Generation

2026-09-21

AI Summary

研究通过创建ChartJudgeBench评估大型多模态模型在图表转代码生成中的裁判能力，揭示了这些模型存在的四个系统性局限。

Abstract

Building strong chart-to-code systems increasingly relies on reinforcement learning, whose effectiveness depends critically on the quality of the reward signal. Large Multimodal Models (LMMs) play a natural critical role in jointly assessing chart visual appearance and task requirements. They are therefore increasingly used as visual critics and reward models, yet their reliability as judges remains largely unexplored. To this end, we introduce ChartJudgeBench, a diagnostic vision-language benchmark for assessing LMM judges in chart-to-code workflows. It includes 1,003 Chart Perception Alignment (CPA) instances for pairwise chart comparison and 650 Chart Reasoning Judgment (CRJ) instances for binary Accept/Reject verification in Chart Reproduction and Chart Editing. Together, these tasks emulate the core judging decisions required in agentic refinement and RL-based chart optimization. Our evaluation of strong LMMs reveals four systematic limitations: (i) positional bias in pairwise comparison, (ii) a strong tendency to overpredict Accept, (iii) difficulty in matching visual styles and aesthetics, and (iv) an unexpected leniency bias in RL-trained models. These findings show that current LMM judges require explicit reliability validation before being used as critics or reward models in chart-to-code optimization. The code and data are available on ChartJudgeBench.

Problem

• Large Multimodal Models
• chart-to-code generation
• reliability
• judges
• reward models

Innovation

• ChartJudgeBench
• Large Multimodal Models (LMMs)
• chart-to-code generation
• reinforcement learning
• visual critics