TokTier: Exact Stateful Tokenization for Agentic LLM Serving

2026-07-31

AI Summary

This work addresses the inefficiency in large language model (LLM) serving caused by repeated tokenization when incrementally appending tool-generated text, which incurs substantial overhead and risks perturbing existing token boundaries, thereby hindering result reuse. To resolve this, the authors propose TokTier—the first strictly faithful incremental tokenization system—that guarantees bit-for-bit equivalence with full re-tokenization while enabling efficient updates through stable boundary detection, localized re-tokenization, and dynamic window adjustment. TokTier integrates GPU-accelerated exact pre-tokenization with Byte Pair Encoding (BPE). When integrated with vLLM, it reduces median time-to-first-token latency by 16–34% and P99 latency by 23%. Incremental fixes for inputs ranging from 100K to 3M characters require only 0.5–1.1 ms—up to 437× faster than HuggingFace tokenizers—and GPU-based full tokenization of 1M characters completes in just 0.87 ms, achieving a 491× speedup.

Abstract

LLM serving systems cache prompt KV state, yet most front ends still re-tokenize the full request text on every call. The cost lands on coding agents, which resubmit a long transcript after each small tool result, and reuse is hard because even a short append can change token boundaries near the end of the previous sequence. Across 153,951 calls from two agent ecosystems, the median call appends about 1.4K characters, and only 1.0-3.6% of calls start or rebuild a session with contexts of millions of characters. At a 94.1% fleet prompt-cache hit rate, tokenization reaches up to 64% of time to first token. TokTier is a stateful tokenization service with one contract: emitted token IDs are always identical to full reference tokenization of the request text. For a session continuation, it re-tokenizes a small window around the append and splices only after a per-request stable-boundary check, widening the window or falling back to full tokenization on failure. For a call without a reusable prefix, it decomposes GPT-family regex pre-tokenization into run-local rules and runs exact pre-tokenization and BPE on a GPU. A sampled shadow verifier re-checks live traffic. Across 17 tokenizer families, differential campaigns cover 1.5x10^10 split checks, a 12.4 TB real-text corpus, and 93,000+ replayed agent steps, with zero divergence. Incremental repair takes 0.5-1.1 ms from 100K to 3M characters, up to 437x faster than HF tokenization and 2.1x faster at 1M than the strongest cache-based baseline (Gigatoken) fully prewarmed. GPU full tokenization encodes a 1M-character request in 0.87 ms, up to 491x below HF and 23.4x below the fastest published CPU method. With vLLM, median time to first token drops 16-34% and P99 drops 23% under recorded bursts. Under a 50 ms P99 objective, four repair cores plus one GPU sustain 1,821 requests/s where a 16-core stateless front end saturates at 40.

Problem

• tokenization
• LLM serving
• stateful tokenization
• agent systems
• prompt caching

Innovation

• stateful tokenization
• incremental tokenization
• GPU-accelerated tokenization
• exact token consistency
• agent LLM serving