ngram frequency analysis

This study systematically investigates critical failure mechanisms in N-gram-based generative retrieval approaches, such as SEAL and MINDER, including document identifier ambiguity, insufficient diversity, and the disproportionate influence of individual identifiers on ranking outcomes. To address these issues, we introduce the first comprehensive taxonomy of failure modes specific to generative retrieval and employ empirical analysis combined with error attribution techniques to dissect flaws in the N-gram generation process. Furthermore, we develop an open-source, interactive visualization tool that enables researchers to intuitively diagnose how identifier generation impacts retrieval performance. Our work substantially enhances the interpretability and controllability of N-gram-based generative retrieval systems, offering both a theoretical foundation and practical resources to advance this emerging paradigm.

Exact n-gram matching and document provenance on internet-scale text (46 TB) face prohibitive storage overhead and computational bottlenecks. Method: We propose the first scalable, lightweight FM-index optimization system, integrating compression and indexing, enabling multithreaded parallel index construction, memory-aware query optimization, and a Web API service architecture. Contribution/Results: Our system reduces index size to 44% of the original text, accelerates indexing by 18×, and cuts memory consumption by 3.2×—enabling full-index construction on a single node in 50 days. We further introduce the first large-scale empirical framework for analyzing data contamination in large language models, revealing up to 40% training-set contamination in widely used benchmarks such as SQuAD. The system—including its API and interactive interface—is open-sourced to support transparent, reproducible contamination monitoring benchmarks.

Despite the dominance of large language models (LLMs), classical n-gram models remain theoretically insightful yet practically limited by fixed context length and scalability. Method: This paper introduces the ∞-gram framework—a theoretically unbounded, real-time n-gram probability estimator operating at 5-trillion-token scale—implemented via a suffix-array–based *infini-gram* engine enabling millisecond inference and dynamic backoff; further supported by distributed n-gram counting and high-compression storage. Contribution/Results: ∞-gram achieves 47% single-step prediction accuracy; significantly reduces neural LLM perplexity; and—crucially—uncovers a systematic inconsistency in long-suffix coherence of machine-generated text, alongside a previously unidentified structural misalignment between Transformer positional encoding and empirical data distributions.

This work investigates the *n*-gram novelty of language model (LM) generations—i.e., the proportion of *n*-grams in generated text absent from the training corpus. To quantify this, we introduce *n*-novelty, a formal metric, and develop Rusty-DAWG: the first Rust implementation of a Directed Acyclic Word Graph (DAWG) enabling O(1) average-case *n*-gram lookup for arbitrary *n*. Using Pythia models and systematic *n*-gram frequency analysis, we empirically demonstrate two key findings: (1) for *n* > 4, LM outputs exhibit *lower* *n*-gram novelty than human-written text, contradicting intuition; and (2) training *n*-gram frequency correlates strongly and negatively with model completion loss. These results challenge assumptions about LM creativity and memorization. We publicly release Rusty-DAWG to support reproducible, scalable analysis of pretraining data provenance and LM memory behavior, providing both a novel methodological tool and empirical grounding for future research on LM generalization and memorization.

To address the inefficiency in resource utilization and the trade-off between accuracy and latency inherent in flat, single-stage retrieval within Retrieval-Augmented Generation (RAG), this paper proposes a coarse-to-fine progressive multi-stage retrieval paradigm. We introduce a novel “granularity–quantity–capacity” tripartite coordination mechanism: initially retrieving a broad, coarse-grained candidate set with low computational capacity; then progressively refining granularity, reducing candidate quantity, and increasing model capacity per stage. This is enabled by dynamic granularity control, capacity-adaptive retriever collaboration, and latency-aware scheduling. Extensive evaluation across multiple RAG benchmarks demonstrates that our method achieves retrieval accuracy on par with state-of-the-art single-stage baselines while reducing end-to-end inference latency by nearly 40%. It thus effectively breaks the efficiency–performance trade-off bottleneck of single-stage retrieval, exhibiting strong generalizability and practical engineering applicability.

This work addresses hallucination errors in end-to-end speech recognition for low-resource languages, which often arise from rare entities and domain-specific terms. Existing retrieval-augmented approaches struggle to balance recognition robustness with inference efficiency. To overcome this, the authors propose a lightweight, purely lexical framework that explicitly models speech recognition errors within a sparse retrieval mechanism. By integrating symmetric text normalization, an error-aware TF-IDF algorithm, and a sparse diagonal penalty matrix derived from historical errors, the method prioritizes the retrieval of correction documents containing high-risk misrecognized terms, thereby suppressing both phoneme-level and recurrent hallucination errors. Evaluated on the FLEURS Persian subset, the approach increases error-aware recall from 53.7% to 90.9% and reduces word error rate from 23.06% to 18.83%, all while introducing negligible inference latency.

Extracting high-frequency n-grams—especially for large n—from massive datasets poses significant challenges in terms of accuracy, efficiency, and determinism. Method: This paper proposes Intergrams, a hardware-aware multi-pass algorithm that exploits the power-law distribution of n-gram frequencies. It generates candidate n-grams from frequent (n−1)-grams, applies frequency-based pruning, and incorporates low-level optimizations to progressively shrink the search space across iterative passes. Theoretical analysis guides algorithm design to ensure exactness and strong scalability. Results: On real-world large-scale datasets, Intergrams achieves 10.3×–33× speedup over the state-of-the-art method. It is the first deterministic approach to break the performance bottleneck for extracting high-frequency n-grams with large n, while guaranteeing correctness and scalability.

This study addresses the challenge of adapting traditional sequence-labeling-based named entity recognition (NER) to the generative paradigm of large language models (LLMs), and presents the first systematic evaluation of open-source LLMs on both flat and nested NER tasks. Through experiments on standard benchmarks using parameter-efficient fine-tuning (e.g., LoRA), structured output formats (inline brackets, XML), and models across multiple scales, the work demonstrates that open-source LLMs can achieve performance comparable to—or even surpassing—that of conventional encoder-based models and GPT-3 when prompted with structured formats. The results indicate that the NER capability of these models stems from their instruction-following and generalization abilities rather than memorization of entity–label pairs, and that fine-tuning has minimal adverse impact on their general capabilities—sometimes even enhancing them.

This study investigates the capacity of small language models (7B parameters or fewer) to effectively leverage external information in retrieval-augmented generation (RAG). Through systematic evaluation on models such as SmolLM2, Qwen2.5, and Llama 3.1—combined with BM25, E5-large-v2, and oracle retrievers across multiple prompt templates—the work introduces a novel parameterized knowledge partitioning framework that cleanly disentangles retrieval failure from context utilization failure for the first time. The findings reveal a fundamental bottleneck in small models’ ability to use retrieved content: even under oracle retrieval conditions, 85%–100% of samples fail to correctly incorporate the relevant answer, and 42%–100% of the model’s original knowledge is disrupted by the retrieved context. The dominant error mode is generation entirely unrelated to the provided context, indicating a pervasive inability to attend to or integrate external information.

This work addresses the vulnerability of generative retrieval under limited beam-width decoding, where relevant documents are often missed due to premature prefix pruning and planning signals that are highly sensitive to query perturbations. The study reproduces and stress-tests the PAG method, which enhances robustness by computing document-level look-ahead priors through synchronized decoding to guide sequence generation. It introduces, for the first time, a diagnostic tool termed “plan drift” to systematically evaluate the impact of surface-form variations in queries on planning stability. Furthermore, the paper explores reindexing-free mitigation strategies for cross-lingual retrieval and query spelling errors. Experiments confirm that PAG is effectively reproducible on MS MARCO and TREC-DL benchmarks; however, intent-preserving misspellings can induce planning collapse, while query translation substantially recovers cross-lingual performance.