Leveraging GenAI for Segmenting and Labeling Centuries-old Technical Documents

2026-02-24

AI Summary

This study addresses the challenges of segmenting and semantically annotating images from 16th–17th century maritime-era shipbuilding manuscripts, which are hindered by extreme data scarcity and the highly specialized nature of the domain. To overcome these obstacles, the work presents a novel integration of SAM2, Florence2, and ChatGPT, augmented with the shipbuilding domain ontology ontoShip and terminology glossary glosShip. This synergistic approach enables accurate automatic image segmentation and semantic annotation tailored to historical technical illustrations. The proposed method significantly improves the precision of processing ancient technical documents, thereby establishing an effective pathway for intelligent retrieval and knowledge extraction from historical maritime literature.

Technology Category

• Computer Vision: Segmentation
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Data Mining & Knowledge Management: Semantic Web

Application Category

• Semantics and Knowledge: Representation, semantic annotation, enhancement, enrichments, access and/or integration of a variety of data on the Web
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Image segmentation and image recognition are well established computational techniques in the broader discipline of image processing. Segmentation allows to locate areas in an image, while recognition identifies specific objects within an image. These techniques have shown remarkable accuracy with modern images, mainly because the amount of training data is vast. Achieving similar accuracy in digitized images of centuries-old documents is more challenging. This difficulty is due to two main reasons: first, the lack of sufficient training data, and second, because the degree of specialization in a given domain. Despite these limitations, the ability to segment and recognize objects in these collections is important for automating the curation, cataloging, and dissemination of knowledge, making the contents of priceless collections accessible to scholars and the general public. In this paper, we report on our ongoing work in segmenting and labeling images pertaining to shipbuilding treatises from the XVI and XVII centuries, a historical period known as the Age of Exploration. To this end, we leverage SAM2 for image segmentation; Florence2 and ChatGPT for labeling; and a specialized ontology ontoShip and glossary glosShip of nautical architecture for enhancing the labeling process. Preliminary results demonstrate the potential of marrying these technologies for improving curation and retrieval of priceless historical documents. We also discuss the challenges and limitations encountered in this approach and ideas on how to overcome them in the future.

Problem

• image segmentation
• historical document analysis
• technical treatises
• data scarcity
• domain specialization

Innovation

• Generative AI
• Image Segmentation
• Domain-specific Ontology
• Historical Document Processing
• Multimodal Labeling