🤖 AI Summary
Existing OCR systems and vision-language models (e.g., ViT) exhibit high vulnerability to visual adversarial perturbations induced by Unicode combining diacritical marks—perturbations that remain imperceptible to humans yet cause severe misrecognition under black-box attacks.
Method: This work introduces the first Unicode-combining-character-based text-to-vision adversarial attack paradigm. It models cross-platform text rendering behavior and devises a genetic-algorithm-driven black-box framework for generating adversarial perturbations—requiring neither model gradients nor training data.
Contribution/Results: Our method successfully compromises production-grade OCR and multimodal models from Facebook, Microsoft, IBM, and Google. 98.7% of generated adversarial samples retain full human readability, while exhibiting strong transferability and practicality. The results expose a critical blind spot in multimodal security: the text rendering layer, which has been largely overlooked in prior adversarial robustness research.
📝 Abstract
While text-based machine learning models that operate on visual inputs of rendered text have become robust against a wide range of existing attacks, we show that they are still vulnerable to visual adversarial examples encoded as text. We use the Unicode functionality of combining diacritical marks to manipulate encoded text so that small visual perturbations appear when the text is rendered. We show how a genetic algorithm can be used to generate visual adversarial examples in a black-box setting, and conduct a user study to establish that the model-fooling adversarial examples do not affect human comprehension. We demonstrate the effectiveness of these attacks in the real world by creating adversarial examples against production models published by Facebook, Microsoft, IBM, and Google.
