Fixing ill-formed UTF-16 strings with SIMD instructions

2026-01-09

arXiv.org

AI Summary

This work addresses the security and reliability risks posed by mismatched surrogate pairs in UTF-16 strings, which commonly arise from data corruption or encoding errors. It presents the first application of SIMD (Single Instruction, Multiple Data) parallel computing techniques to UTF-16 validation and repair, enabling efficient identification and replacement of invalid surrogate pairs with the Unicode replacement character U+FFFD. By processing multiple 16-bit code units in parallel, the proposed method achieves substantial improvements in throughput and efficiency on modern CPUs. The algorithm has been integrated into the Google V8 JavaScript engine and is now widely deployed in mainstream web browsers, significantly enhancing string processing performance.

Technology Category

• Search and Optimization: Distributed Search
• Natural Language Processing: Safety and Robustness
• Application Domains: Humanities & Computational Social Science

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

UTF-16 is a widely used Unicode encoding representing characters with one or two 16-bit code units. The format relies on surrogate pairs to encode characters beyond the Basic Multilingual Plane, requiring a high surrogate followed by a low surrogate. Ill-formed UTF-16 strings -- where surrogates are mismatched -- can arise from data corruption or improper encoding, posing security and reliability risks. Consequently, programming languages such as JavaScript include functions to fix ill-formed UTF-16 strings by replacing mismatched surrogates with the Unicode replacement character (U+FFFD). We propose using Single Instruction, Multiple Data (SIMD) instructions to handle multiple code units in parallel, enabling faster and more efficient execution. Our software is part of the Google JavaScript engine (V8) and thus part of several major Web browsers.

Problem

• UTF-16
• ill-formed strings
• surrogate pairs
• Unicode replacement character
• data corruption

Innovation

• SIMD
• UTF-16
• surrogate pairs
• ill-formed string repair
• Unicode replacement