Multilingual OCR-Aware Fine-Tuning and Prompt-Guided Chain-of-Thought Reasoning for Multimodal Large Language Models

2026-05-13

AI Summary

This work addresses the poor performance of multimodal large language models (MLLMs) on real-world images, where challenges such as low-quality optical character recognition (OCR)—caused by tiny, blurry, occluded, or complexly formatted text—and inadequate multilingual understanding often lead to failure. To overcome these limitations, the authors propose the first training framework that integrates OCR-aware fine-tuning, large-scale synthetic multilingual OCR-translation data, and structured visual chain-of-thought (CoT) prompting, built upon the LLaMA architecture with efficient LoRA adaptation. This approach significantly enhances model robustness to degraded text, reduces reliance on linguistic priors and hallucination, and outperforms existing baselines—including GPT-5-class and Gemini-series models—on multilingual receipts, menus, and posters, demonstrating superior OCR grounding and text extraction accuracy.

Abstract

Optical character recognition (OCR) and multilingual text understanding remain major failure modes of multimodal large language models (MLLMs), particularly in real-world images containing cluttered layouts, small fonts, blur, occlusion, and complex typography. We present an OCR-aware multilingual multimodal training framework that combines (i) large-scale synthetic OCR-to-translation data generation, (ii) OCR-aware supervised fine-tuning (SFT) with LoRA adaptation, and (iii) structured visual chain-of-thought (CoT) prompting for reasoning under uncertain visual conditions. Using a LLaMA-based multimodal architecture, the proposed framework substantially improves OCR completeness, multilingual translation accuracy, and robustness under degraded visual conditions. Experimental results on multilingual receipts, menus, posters, signs, handwritten text, and document images demonstrate significantly improved visual-text grounding compared with the baseline model. In particular, the proposed OCR-aware post-training framework improves extraction of small, blurred, spatially scattered, and partially occluded text while reducing reliance on language priors under uncertain OCR conditions. Qualitative comparisons with frontier multimodal systems, including GPT-5-class and Gemini-family models, further suggest improved OCR grounding and reduced hallucination under noisy and visually ambiguous OCR scenarios. Overall, the results indicate that data-centric OCR-aware multimodal post-training provides an effective and scalable direction for improving multilingual OCR and OCR-based visual question answering systems.

Problem

• multilingual OCR
• multimodal large language models
• text understanding
• visual-text grounding
• degraded visual conditions

Innovation

• OCR-aware fine-tuning
• multilingual multimodal learning
• chain-of-thought reasoning
• LoRA adaptation
• visual-text grounding