BanglaWild: An In-the-Wild Bengali Scene Text Recognition Benchmark for OCR and Vision-Language Models

2026-08-04

AI Summary

This study addresses the lack of a unified benchmark for evaluating Bengali text recognition in real-world scenarios and the absence of joint assessment between traditional OCR systems and vision-language models (VLMs). To bridge this gap, the authors introduce BanglaWild, a benchmark comprising 2,535 real-world images annotated with precise transcriptions, category labels, diagnostic attributes, and standardized spellings. For the first time, they systematically evaluate 15 VLMs and 3 OCR systems on a common dataset, employing LoRA fine-tuning and an LLM-as-a-Judge evaluation strategy. Key contributions include a multidimensional annotation framework and a 15-category error taxonomy. The analysis reveals that approximately 60% of errors stem from visual misrecognition rather than conjunct character errors (<2%), challenges the assumption that larger models within the same family consistently outperform smaller ones, and demonstrates that LoRA fine-tuning mitigates failures in weaker models but offers limited gains for already strong models.

Abstract

In-the-wild Bengali scene text recognition is largely unmeasured: existing resources target handwritten documents or constrained sign-board parsing, report only aggregate edit-distance metrics, and evaluate either conventional OCR or VLMs, never both on the same in-the-wild data. To address this gap, we introduce BANGLAWILD, a benchmark of 2,535 Bengali scene text images, each paired with a verbatim gold transcription, two categorical axes, four diagnostic attributes, and an orthographically standard form where the in-image text deviates from canonical spelling. We evaluate fifteen VLMs and three conventional OCR systems under three prompting strategies, fine-tune 6 open-source models with LoRA, and complement edit-distance metrics with an LLM-as-a-Judge evaluation. Our results reveal a persistent gap in which larger models within the same family do not outperform smaller ones. Our fifteen-class error taxonomy shows that visual mis-recognition accounts for ~60% of errors in the strongest systems, while conjunct-related errors contribute under 2%, challenging a long-standing assumption in Bengali OCR research; the same visual dominant profile also holds across architectures, including the one conventional baseline that reads Bengali reliably. Prompt language mainly affects cross-script drift and LoRA reduces catastrophic failures in weak models without lifting the ceiling on already competent ones. Code and data will be publicly released.

Problem

• Bengali scene text recognition
• in-the-wild OCR
• vision-language models
• benchmarking
• text recognition evaluation

Innovation

• BanglaWild
• scene text recognition
• vision-language models
• LLM-as-a-Judge
• error taxonomy