Page image classifier fine-tuned on century-spanning archives of scanned documents for further content-specific processing

2026-05-25

AI Summary

This study addresses the challenge of automatically classifying large-scale scanned historical document pages containing diverse visual elements such as text, tables, and figures. The authors construct a dataset of over 48,000 annotated images and collaborate with domain experts to define an 11-category visual content taxonomy. They systematically evaluate multiple models, including a handcrafted-feature-based random forest baseline, fine-tuned EfficientNetV2, RegNetY, Vision Transformer (ViT), DiT, and multimodal CLIP. Among these, RegNetY-16GF achieves a Top-1 accuracy of 99.16% and demonstrates over 90% prediction agreement with other models on more than 640,000 unlabeled pages, substantially outperforming CLIP. This work presents the first systematic comparison of CNNs, Transformers, and CLIP in terms of deployment consistency on large-scale historical documents, providing high-accuracy classification to support downstream OCR and structural analysis.

Abstract

Purpose: Digitization projects in the humanities produce vast, heterogeneous archives of historical documents, making manual sorting impractical at scale. This work addresses the need for an automated system to classify scanned page images based on visual content type - text, tables, and graphics - enabling content-specific downstream processing such as Optical Character Recognition (OCR) or structured data extraction. Methods: An image classification system was developed and evaluated on a dataset of over 48,000 annotated historical page images from century-old Czech archaeological archives, refined through four successive annotation stages with domain-expert review. A Random Forest Classifier baseline was established using hand-crafted image features. Subsequently, deep learning architectures were fine-tuned and compared: Convolutional Neural Networks (EfficientNetV2, RegNetY), Vision and Document Image Transformers (ViT, DiT), and multimodal CLIP models. An 11-category label scheme was designed collaboratively with domain experts and evaluated via five-fold cross-validation. Results: The feature-based baseline achieved approximately 75% accuracy. Fine-tuned CNNs and Transformers substantially outperformed it, with RegNetY-16GF achieving 99.16% and ViT-large 99.12% Top-1 accuracy on the held-out test set. CLIP ViT-B/16 reached 99.14% with optimized text descriptions. Conclusion: Image-only models, particularly RegNetY-16GF, deliver near-perfect classification accuracy and produce consistent labels across 649,508 unlabeled archival pages with over 90% inter-model agreement. Fine-tuned CLIP, despite competitive test-set accuracy, showed under 65% agreement with image-only models on unlabeled data, making it less suitable for deployment. The final models, annotated dataset, and software are publicly available under open-source licenses.

Problem

• page image classification
• historical document analysis
• content type recognition
• automated document processing
• visual content categorization

Innovation

• fine-tuned deep learning
• historical document image classification
• RegNetY
• Vision Transformer (ViT)
• content-specific processing