Logographic Character Visual Pretraining via Semantic-based Contrastive Learning

2026-07-30

AI Summary

This work addresses the performance bottleneck in visual tasks involving logographic scripts—such as Chinese—caused by imbalanced data distributions and scarce samples of rare characters. To overcome the limitations of purely vision-based approaches, the authors propose a multimodal contrastive learning pretraining method that integrates character-level visual features with contextual semantics derived from language models. This is the first study to incorporate context-aware semantic representations into the visual representation learning of logographic characters, introducing a novel semantic alignment strategy tailored for highly imbalanced datasets. Experimental results demonstrate that the proposed approach significantly outperforms state-of-the-art models across multiple character recognition benchmarks and downstream tasks, with particularly notable improvements in recognizing rare and low-frequency characters.

Abstract

Current deep learning-based character vision studies, e.g., text recognition, character image denoising, and historical text completion, are offering new solutions for learning, managing, and utilizing character resources. However, the performance of these studies peaks only with large and balanced datasets, which is a rarity with real-world character datasets, especially for logographic character languages, e.g., Chinese. The imbalance in data distribution of logographic characters is a common issue due to differences in character usage frequency and new characters being continuously created. In this paper, we propose a novel method for logographic character recognition, which introduces a multi-modal learning approach using visual semantics and contextual semantics of characters. A novel pre-training strategy is designed to enhance deep visual representations, especially for datasets suffering from issues of imbalanced and rare instances, by extracting the contextual semantics of each character from the corresponding language models. We conduct experiments across various datasets to evaluate our character recognition method and further validate the contrastive pre-training strategy by several downstream tasks. Experimental results demonstrate the superiority of our method compared to state-of-the-art methods.

Problem

• logographic characters
• data imbalance
• character recognition
• visual pretraining
• rare instances

Innovation

• logographic character recognition
• semantic-based contrastive learning
• multi-modal pretraining
• visual semantics
• contextual semantics