CORU: Comprehensive Post-OCR Parsing and Receipt Understanding Dataset

2024-06-06

arXiv.org

AI Summary

Multilingual receipt understanding—particularly for Arabic—faces significant challenges due to complex layouts, high noise levels, and low cross-lingual OCR accuracy. To address these, we introduce CORU, the first bilingual Arabic–English receipt understanding dataset, comprising over 20,000 structured receipts, 30,000 OCR-line-level images, and 10,000 fine-grained item-level annotations, all drawn from real-world retail scenarios. CORU supports three core tasks: object detection, OCR, and information extraction. We propose a unified multimodal annotation framework and develop a reproducible end-to-end parsing baseline integrating Tesseract, YOLO-based layout analysis, sequence labeling, and multimodal information extraction models. Experiments demonstrate substantial improvements in Arabic text recognition accuracy and F1 scores for key fields (e.g., total amount, item names). CORU is publicly released to serve as a benchmark for multilingual receipt understanding research.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

In the fields of Optical Character Recognition (OCR) and Natural Language Processing (NLP), integrating multilingual capabilities remains a critical challenge, especially when considering languages with complex scripts such as Arabic. This paper introduces the Comprehensive Post-OCR Parsing and Receipt Understanding Dataset (CORU), a novel dataset specifically designed to enhance OCR and information extraction from receipts in multilingual contexts involving Arabic and English. CORU consists of over 20,000 annotated receipts from diverse retail settings, including supermarkets and clothing stores, alongside 30,000 annotated images for OCR that were utilized to recognize each detected line, and 10,000 items annotated for detailed information extraction. These annotations capture essential details such as merchant names, item descriptions, total prices, receipt numbers, and dates. They are structured to support three primary computational tasks: object detection, OCR, and information extraction. We establish the baseline performance for a range of models on CORU to evaluate the effectiveness of traditional methods, like Tesseract OCR, and more advanced neural network-based approaches. These baselines are crucial for processing the complex and noisy document layouts typical of real-world receipts and for advancing the state of automated multilingual document processing. Our datasets are publicly accessible (https://github.com/Update-For-Integrated-Business-AI/CORU).

Problem

• Multilingual OCR for Arabic-English receipts
• Information extraction from complex receipt layouts
• Dataset supports object detection and QA tasks

Innovation

• Multilingual OCR dataset for Arabic-English receipts
• Comprehensive annotations for object detection and extraction
• Baselines with Tesseract and advanced neural networks