Component-Weighted Centroid Search for Exact Incremental BPE

2026-09-30

AI Summary

This study addresses the O(log²t) worst-case time complexity of centroid search in incremental BPE algorithms by proposing a recursive component-size weighting strategy. Leveraging cost-scaling principles to optimize local search, this method reduces the worst-case complexity of a single append operation to O(log t) while preserving semantic equivalence. A prototype system implemented in Rust, combined with a normalized correct-merging model, provides formal theoretical verification. Experimental results demonstrate that the empirically measured probe counts of the weighted search align with theoretical predictions and outperform count-balanced search under specific constructions, thereby establishing superior worst-case performance guarantees.

Abstract

Exact incremental BPE maintains the canonical tokenization state after every appended byte. The recent algorithm of Jiang and Gong (2026) does this in $O(\log^2 t)$ worst-case time, where $t$ is the maximum canonical token length. Its centroid search visits $O(\log t)$ components and can pay another $O(\log t)$ for ordered point location at each one. Within Jiang and Gong's normalized/proper merge-stage model, we change only that local search. Each interval is weighted by the size of the recursive component it selects, so a move from size $m$ to size $m'$ costs $O(1+\log(m/m'))$. These charges telescope, giving $O(\log t)$ time per append and $O(n\log t)$ over an $n$-byte stream, with the same BPE semantics and asymptotic space. We also construct a normalized proper BPE family over a fixed alphabet where count-balanced search uses $Θ(\log^2 t)$ probes on a reachable update, while the weighted search uses $Θ(\log t)$. A Rust implementation matches the predicted probe counts on every tested instance. On ordinary vocabularies the queried degrees are small, however, and the improvement is a worst-case guarantee rather than an average-speed result.

Problem

• Incremental BPE
• centroid search
• worst-case time complexity
• byte pair encoding

Innovation

• Incremental BPE
• Centroid Search
• Component-Weighted Search
• Time Complexity
• Tokenization