Error Patterns in Historical OCR: A Comparative Analysis of TrOCR and a Vision-Language Model

2026-02-16

AI Summary

This study addresses the challenges of optical character recognition (OCR) for eighteenth-century printed texts, which include archaic glyphs, non-standardized spelling, and print degradation—factors that render conventional evaluation metrics inadequate for assessing scholarly reliability. The authors propose a combined approach using length-weighted accuracy and hypothesis-driven error analysis to systematically compare the performance of the specialized OCR model TrOCR against the general-purpose vision-language model Qwen on historical English text lines. Their findings reveal systematic differences rooted in architectural inductive biases: Qwen exhibits lower overall error rates and greater robustness to degraded inputs but implicitly normalizes spelling, whereas TrOCR more faithfully preserves original orthography at the cost of susceptibility to cascading errors. The work underscores the necessity of aligning model selection with scholarly risk assessment in the digitization of historical documents.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Optical Character Recognition (OCR) of eighteenth-century printed texts remains challenging due to degraded print quality, archaic glyphs, and non-standardized orthography. Although transformer-based OCR systems and Vision-Language Models (VLMs) achieve strong aggregate accuracy, metrics such as Character Error Rate (CER) and Word Error Rate (WER) provide limited insight into their reliability for scholarly use. We compare a dedicated OCR transformer (TrOCR) and a general-purpose Vision-Language Model (Qwen) on line-level historical English texts using length-weighted accuracy metrics and hypothesis driven error analysis. While Qwen achieves lower CER/WER and greater robustness to degraded input, it exhibits selective linguistic regularization and orthographic normalization that may silently alter historically meaningful forms. TrOCR preserves orthographic fidelity more consistently but is more prone to cascading error propagation. Our findings show that architectural inductive biases shape OCR error structure in systematic ways. Models with similar aggregate accuracy can differ substantially in error locality, detectability, and downstream scholarly risk, underscoring the need for architecture-aware evaluation in historical digitization workflows.

Problem

• Historical OCR
• Error Patterns
• Orthographic Fidelity
• Scholarly Reliability
• Character Error Rate

Innovation

• historical OCR
• error analysis
• Vision-Language Model
• TrOCR
• orthographic fidelity