TABVERSE: Benchmarking Cross-Format Table Understanding in LLMs and VLMs

2026-06-08

AI Summary

Existing benchmarks for table understanding struggle to disentangle the intertwined effects of content, formatting, and modality. To address this limitation, this work introduces TABVERSE—a controlled, multimodal benchmark that rigorously aligns identical table content across multiple representations, including HTML, Markdown, LaTeX, and rendered images, with annotations for question categories and difficulty levels. By fixing content while varying representation formats, TABVERSE enables systematic evaluation of large language models (LLMs) and vision-language models (VLMs) on three core tasks: question answering, structural comprehension, and table reconstruction. Experimental results reveal that models consistently outperform on structured textual formats—particularly HTML—compared to images. Performance is jointly influenced by task type, model architecture, and format choice, with row-sensitive reasoning and LaTeX-based reconstruction remaining notably challenging.

Abstract

Large Language Models (LLMs) and Vision-Language Models (VLMs) are increasingly evaluated on table reasoning tasks, but the role of table representation remains under-explored. In practice, the same table content may appear in different structural formats, such as HTML, Markdown, and LaTeX, or as rendered images. However, existing evaluations often let content, format, layout, and modality vary together, making it difficult to isolate representation effects. We introduce TABVERSE, a controlled multimodal table benchmark that aligns the same table content across multiple structural formats and rendered images, with question category and difficulty tags. This design enables systematic evaluation of representation effects while holding table content fixed. We evaluate LLMs and VLMs across three tasks: Question Answering (QA), Structural Understanding Capability (SUC), and Structure Reconstruction (SR). Our results show that representation choice substantially affects table understanding. Models generally perform better with structured text than with rendered images, but the size of this gap depends on the task, model, and format. HTML is often the most robust text format, while row-sensitive structural tasks and syntactically usable LaTeX reconstruction remain challenging. These findings show that table representation is a key factor in reliable table evaluation.

Problem

• table understanding
• representation format
• multimodal benchmark
• structured data
• model evaluation

Innovation

• table representation
• multimodal benchmark
• controlled evaluation
• structured format
• vision-language models