TFLOP: Table Structure Recognition Framework with Layout Pointer Mechanism

2024-08-01

International Joint Conference on Artificial Intelligence

AI Summary

Existing table structure recognition methods suffer from text-position misalignment in complex tables, watermarked documents, and multilingual settings, relying on cascaded text detection and post-hoc alignment—limiting robustness. This paper proposes Layout Pointer, an end-to-end framework that unifies structural parsing and text localization into a joint pointer prediction task, eliminating traditional two-stage pipelines. We introduce a novel span-aware contrastive supervision mechanism to enhance modeling capability under cross-lingual and noisy conditions. Built upon a Transformer architecture, the model enables direct, post-processing-free joint decoding of structure and text. It achieves state-of-the-art performance on PubTabNet, FinTabNet, and SynthTabNet, and significantly outperforms prior methods on real-world industrial documents—including those with watermarks and non-English content. The code is publicly available.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Structured Learning
• Computer Vision: Language and Vision

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Table Structure Recognition (TSR) is a task aimed at converting table images into a machine-readable format (e.g. HTML), to facilitate other applications such as information retrieval. Recent works tackle this problem by identifying the HTML tags and text regions, where the latter is used for text extraction from the table document. These works however, suffer from misalignment issues when mapping text into the identified text regions. In this paper, we introduce a new TSR framework, called TFLOP (TSR Framework with LayOut Pointer mechanism), which reformulates the conventional text region prediction and matching into a direct text region pointing problem. Specifically, TFLOP utilizes text region information to identify both the table's structure tags and its aligned text regions, simultaneously. Without the need for region prediction and alignment, TFLOP circumvents the additional text region matching stage, which requires finely-calibrated post-processing. TFLOP also employs span-aware contrastive supervision to enhance the pointing mechanism in tables with complex structure. As a result, TFLOP achieves the state-of-the-art performance across multiple benchmarks such as PubTabNet, FinTabNet, and SynthTabNet. In our extensive experiments, TFLOP not only exhibits competitive performance but also shows promising results on industrial document TSR scenarios such as documents with watermarks or in non-English domain. Source code of our work is publicly available at: https://github.com/UpstageAI/TFLOP.

Problem

• Table Recognition
• Complex Table Structure
• Non-English Document

Innovation

• Table Structure Recognition
• Positional Accuracy
• Watermark and Non-English Document Processing