Compiling Rewrite Rules to Finite-State Transducers with the Worsening Trick

2026-06-08

AI Summary

This study addresses the challenge of efficiently compiling general rewrite rules with arbitrary regular-language context constraints into finite-state transducers, particularly in the presence of overlapping matches and complex contextual dependencies. The work proposes a compact compilation method based on a “worsening trick”: it first generates all valid rewrite candidates and then filters out suboptimal ones that admit better alternatives. This approach uniformly supports multiple contexts, arbitrary transformations, flag diacritics, directional rewriting, weighted rules, and parallel application. Formulated within a concise formal framework, the method reproduces classical results under semantic equivalence while significantly improving scalability and implementation simplicity. Experimental evaluation demonstrates that the generated transducers are functionally equivalent to those produced by foma across extensive grammatical and regression test suites, differing only in state numbering, thereby confirming both correctness and practical utility.

Abstract

Finite-state transducers (FSTs) are essential for modeling string rewriting in computational linguistics and natural language processing (NLP), particularly for phonological and morphological rewrite rules. Compiling general rewrite rules of the form $A \to B / L \, \_ \, R$, where $A$, $B$, $L$, and $R$ are arbitrary regular languages, is complex due to overlapping matches and context constraints. Traditional methods, such as those by Kaplan and Kay or Karttunen, rely on intricate transducer compositions with auxiliary markers. This paper presents a compact compilation scheme based on the "worsening trick'': generate all legal rewrite candidates, then filter candidates that are worse than another candidate for the same input. Implemented as the built-in rewrite compiler in PyFoma, the construction supports multiple contexts, arbitrary transductions, markup, directed rewriting, weights, and parallel rewriting. The resulting formulas are short and uniform, and where semantics coincide, they reproduce the same rule transducers as earlier approaches while remaining easier to extend. The implementation has been validated against foma on both a substantial collection of rewrite grammars and an automated regression suite covering the major rewrite modalities, with the resulting transducers matching exactly apart from state numbering.

Problem

• rewrite rules
• finite-state transducers
• context constraints
• overlapping matches
• compilation

Innovation

• finite-state transducers
• rewrite rules
• worsening trick
• compilation
• PyFoma