🤖 AI Summary
This work investigates the robustness of multimodal large language models (MLLMs), such as Gemini-1.5-pro, to perspective distortion in document structure extraction. To address performance degradation caused by non-frontal capture angles in real-world scenarios, we model perspective distortion using an isosceles trapezoidal transformation, compressing the eight-dimensional distortion space into two interpretable geometric parameters—rotation angle and distortion ratio—for systematic distortion control. Using a synthetic document dataset, we quantitatively evaluate character-level and structure-level recognition accuracy of both OCR systems and MLLMs, revealing high sensitivity to distortion. We further demonstrate that lightweight rotation-based rectification significantly improves performance. This study establishes, for the first time, a quantitative relationship between perspective distortion and recognition accuracy. It provides a reproducible evaluation framework and practical rectification strategies to enhance the practicality of MLLMs under suboptimal imaging conditions.
📝 Abstract
Optical Character Recognition (OCR) for data extraction from documents is essential to intelligent informatics, such as digitizing medical records and recognizing road signs. Multi-modal Large Language Models (LLMs) can solve this task and have shown remarkable performance. Recently, it has been noticed that the accuracy of data extraction by multi-modal LLMs can be affected when in-plane rotations are present in the documents. However, real-world document images are usually not only in-plane rotated but also perspectively distorted. This study investigates the impacts of such perturbations on the data extraction accuracy for the state-of-the-art model, Gemini-1.5-pro. Because perspective distortions have a high degree of freedom, designing experiments in the same manner as single-parametric rotations is difficult. We observed typical distortions of document images and showed that most of them approximately follow an isosceles-trapezoidal transformation, which allows us to evaluate distortions with a small number of parameters. We were able to reduce the number of independent parameters from eight to two, i.e. rotation angle and distortion ratio. Then, specific entities were extracted from synthetically generated sample documents with varying these parameters. As the performance of LLMs, we evaluated not only a character-recognition accuracy but also a structure-recognition accuracy. Whereas the former represents the classical indicators for optical character recognition, the latter is related to the correctness of reading order. In particular, the structure-recognition accuracy was found to be significantly degraded by document distortion. In addition, we found that this accuracy can be improved by a simple rotational correction. This insight will contribute to the practical use of multi-modal LLMs for OCR tasks.