🤖 AI Summary
This paper addresses efficient regular expression parsing and submatch extraction by proposing a unified framework based on Tagged Deterministic Finite Automata (TDFA). Methodologically, it presents for the first time a complete, systematic treatment of TDFA construction, optimization, and ambiguity resolution—including POSIX and leftmost-longest semantics—via rigorous pseudocode and step-by-step examples; supports both precompilation and on-the-fly determinization; and incorporates production-grade optimizations such as state merging and deferred transitions. Contributions include: (1) the first fully implemented TDFA solution supporting multiple semantic policies, dual determinization modes, and end-to-end optimization; and (2) empirical validation via both the RE2C generator and a standalone Java library, demonstrating substantial speedups over conventional NFA/DFA approaches across multiple benchmark suites—achieving state-of-the-art runtime performance.
📝 Abstract
We present an algorithm for regular expression parsing and submatch extraction based on tagged deterministic finite automata. The algorithm works with different disambiguation policies. We give detailed pseudocode for the algorithm, covering important practical optimizations. All transformations from a regular expression to an optimized automaton are explained on a step-by-step example. We consider both ahead-of-time and just-in-time determinization and describe variants of the algorithm suited to each setting. We provide benchmarks showing that the algorithm is very fast in practice. Our research is based on two independent implementations: an open-source lexer generator RE2C and an experimental Java library.