🤖 AI Summary
This study addresses the lack of systematic, quantitative analysis regarding how document layout markers—such as paragraph boundary delimiters—in pretraining corpora influence language model behavior. The authors propose a novel “clean-window survival” metric and employ preregistered experiments, multi-model prediction difficulty assessments, and a reversible sidecar data format to empirically evaluate 13 public corpora. Their findings demonstrate that the critical factor affecting model predictions is not the specific delimiter symbols themselves, but rather the structural announcement information they convey. Removing these structural announcements substantially degrades model performance, whereas substituting the delimiter symbols has negligible impact. Moreover, models cannot spontaneously reconstruct missing structural cues. This work also provides the first quantitative characterization of layout marker distributions and introduces a “pure-frame” data format to disentangle markup from textual content.
📝 Abstract
How a document's arrangement is written down, its notation, is a training variable that no dataset card records. The field has established that text-extraction choices change model behaviour, and has never once measured the notation of what those choices put into the corpus. We define clean-window survival, a deterministic count of how much of a stream still demands the boundary inference, and measure notation on three fronts. What corpora carry: a census of thirteen public corpora, where survival falls to 0.153 in a vision-converted PDF slice against 0.889 in C4; the scarce resource is not unmarked text but long unmarked text; a pre-registered supply test finds what remains institutional, not consumer. Our own pre-registered prediction failed: converters do not fabricate structure on prose, and that null forced the reliability mechanism that survives it. What readers use: across five base models spanning 0.6B to 8.2B and two pipelines, deleting a structural announcement makes the following prose measurably harder to predict, while swapping its notation moves nothing. That zero does not make notation unimportant; it relocates the variable: the operative cue is the announcement, not the sigil. What writers impose: a bounded null. Base models do not impose the marked register above the authored baseline, and handed prose with every announcement deleted they do not put one back, at a rate indistinguishable from zero against an authored reference of zero. We ship the format those measurements imply: the pure frame, paragraphs in authored order, every announcement deleted into a reversible sidecar, mixed against the marked copy over announcement presence rather than notation. Choose format operators by the capability they train, not by the fidelity they preserve, and record extractor identity and survival on data cards.