optical character recognition

To address the limited flexibility and weak semantic understanding in information extraction from mixed scanned images and natively digital documents, this paper proposes a two-stage OCR-LLM collaborative framework. In the first stage, multi-engine OCR combined with layout-aware parsing (e.g., PDFMiner/LayoutParser) performs preliminary text and structural extraction. In the second stage, a fine-tuned large language model (LLM), enhanced by context-aware prompt engineering, achieves cross-format, multi-layout semantic parsing of key fields and generates confidence scores. The work introduces the novel “OCR pre-extraction + LLM post-parsing” paradigm, integrating visual layout cues with linguistic context to resolve format-induced ambiguities. Experiments demonstrate a 32% improvement in key-field extraction accuracy, support for over ten document types, a 27% reduction in average latency, and significantly enhanced layout robustness and generalization capability.

Vision-language models (VLMs) suffer from low accuracy and poor generalization in business document chart understanding due to inherent visual recognition limitations. Method: We propose a text-only paradigm for chart structure understanding—bypassing image-based analysis entirely and instead parsing structured metadata (e.g., shapes, connectors) directly from editable source files (XLSX/PPTX/DOCX) at the Office Open XML (OOXML) level, then feeding this structured input to large language models (LLMs) for relational reasoning and question answering. Contribution/Results: By eliminating VLMs’ visual bottlenecks and leveraging fine-grained XML parsing with structure-aware prompt engineering, our approach achieves high-precision semantic parsing. On system design document QA tasks, it significantly outperforms VLM baselines. Robust cross-format generalization is validated across PPTX, DOCX, and XLSX, demonstrating strong adaptability to real-world business scenarios. This work establishes a new, interpretable, cost-effective, and high-accuracy pathway for document intelligence.

Traditional OCR systems focus solely on text recognition and struggle to interpret graphical elements such as charts and tables, resulting in significant loss of semantic information in document understanding. This work proposes dots.mocr, a novel approach that treats graphical components as first-class parsing targets alongside text, enabling unified modeling and end-to-end generation of structured textual representations for multimodal documents. Leveraging a large-scale data engine built from PDFs, web pages, and SVGs, the method employs staged pretraining followed by supervised fine-tuning to train a 3-billion-parameter model. Evaluated on the olmOCR Bench, dots.mocr achieves a new state-of-the-art score of 83.9 and ranks second only to Gemini 3 Pro in the OCR Arena, while notably surpassing Gemini 3 Pro in the quality of generated SVG outputs from graphical content.

This study addresses the challenges of multilingual image-document information extraction and weak support for low-resource languages by proposing an end-to-end, lightweight multi-task framework. Methodologically, it introduces the first OCR–large language model (LLM) collaborative pipeline—integrating Tesseract-based cross-lingual OCR, machine translation, abstractive summarization, back-translation, TensorFlow-powered sentiment analysis, Hugging Face Transformers-based topic classification, and rule-based date extraction. The system supports English, Hindi, Tamil, and other languages, achieving high-accuracy cross-lingual understanding and generation on real-world document images. Key contributions include: (1) the first lightweight OCR–LLM co-processing architecture explicitly designed for low-resource languages; (2) significant improvements in document semantic accessibility via joint multi-task optimization; and (3) an interactive Gradio interface that bridges linguistic divides and enhances information access for multilingual users.

To address low OCR accuracy caused by degradation in historical document images, this paper proposes a two-stage end-to-end optimization framework. In the first stage, a U-Net–based image restoration model is trained on a synthetically generated multi-degradation dataset to jointly optimize visual clarity and linguistic consistency. In the second stage, a semantic-aware ByT5 model performs post-OCR error correction, enhanced by a multi-directional block extraction and fusion mechanism tailored for large-format documents. The key innovations include the first joint optimization of image restoration quality and text semantic consistency, and the construction of the first cross-lingual (English/French/Spanish) synthetic dataset for historical text. Evaluated on 13,831 pages of real historical documents, the framework reduces character error rate by 63.9–70.3% over baseline OCR systems, demonstrating substantial improvement.

This study investigates whether document information extraction in the era of multimodal large language models (MLLMs) still necessitates traditional OCR preprocessing. Through large-scale benchmarking, the authors evaluate the end-to-end performance of off-the-shelf MLLMs on real-world business documents and propose a direct image input pipeline that bypasses OCR entirely. They develop an LLM-based automated hierarchical error analysis framework to systematically diagnose failure modes and introduce structured prompting, in-context learning, and output schema constraints to significantly enhance extraction accuracy. Experimental results demonstrate that the OCR-free approach achieves comparable accuracy to OCR-augmented pipelines across most scenarios, confirming the feasibility of streamlining the extraction workflow and offering practical guidance for real-world deployment.

This study addresses the lack of systematic evaluation of robustness against visual perturbations in current vision-language models (VLMs) on structure-sensitive OCR reasoning tasks. The authors introduce OCR-Robust, a benchmark comprising 812 samples from documents, charts, and other structured formats, augmented with five perturbation types at three intensity levels each. They propose a multi-dimensional evaluation framework—including clean accuracy, Robustness under Clean Reference (RCR), Worst-case Clean Reference (WCR), and Clean Robustness Index (CRI)—to conduct the first comprehensive assessment of 18 state-of-the-art VLMs and OCR+LLM pipelines. The findings reveal that high clean accuracy does not necessarily imply strong robustness; inputs involving charts and tables are significantly more vulnerable than plain documents, with some models exhibiting severe performance degradation under worst-case perturbations.

This work investigates the robustness of multimodal large language models (MLLMs), such as Gemini-1.5-pro, to perspective distortion in document structure extraction. To address performance degradation caused by non-frontal capture angles in real-world scenarios, we model perspective distortion using an isosceles trapezoidal transformation, compressing the eight-dimensional distortion space into two interpretable geometric parameters—rotation angle and distortion ratio—for systematic distortion control. Using a synthetic document dataset, we quantitatively evaluate character-level and structure-level recognition accuracy of both OCR systems and MLLMs, revealing high sensitivity to distortion. We further demonstrate that lightweight rotation-based rectification significantly improves performance. This study establishes, for the first time, a quantitative relationship between perspective distortion and recognition accuracy. It provides a reproducible evaluation framework and practical rectification strategies to enhance the practicality of MLLMs under suboptimal imaging conditions.

When BPMN process diagram source files (e.g., XML) are unavailable, recovering structured semantic information directly from raster images remains challenging. Method: We propose an end-to-end vision-language joint approach that tightly integrates multimodal large models (VLMs) with optical character recognition (OCR) via prompt engineering—enabling unified modeling of graphical symbol recognition, text localization, and semantic alignment without manual annotations or textual priors. Contribution/Results: Ablation studies and statistical analysis across multiple VLM benchmarks demonstrate that OCR enhancement significantly improves node-type identification and control-flow relation extraction accuracy (average +12.7%). The method exhibits strong robustness against image degradation—including blurriness, scaling artifacts, and low resolution. This work establishes a practical, deployable paradigm for structured image parsing in reverse engineering and legacy system digitization.

This work addresses the poor performance of multimodal large language models (MLLMs) on real-world images, where challenges such as low-quality optical character recognition (OCR)—caused by tiny, blurry, occluded, or complexly formatted text—and inadequate multilingual understanding often lead to failure. To overcome these limitations, the authors propose the first training framework that integrates OCR-aware fine-tuning, large-scale synthetic multilingual OCR-translation data, and structured visual chain-of-thought (CoT) prompting, built upon the LLaMA architecture with efficient LoRA adaptation. This approach significantly enhances model robustness to degraded text, reduces reliance on linguistic priors and hallucination, and outperforms existing baselines—including GPT-5-class and Gemini-series models—on multilingual receipts, menus, and posters, demonstrating superior OCR grounding and text extraction accuracy.