TempTest: Local Normalization Distortion and the Detection of Machine-generated Text

2025-03-26

AI Summary

Existing zero-shot detection methods rely on statistical measures—such as log-likelihoods, ranking scores, and entropy—derived from language model outputs; however, their discriminative power saturates as models better approximate human text distributions. To address this limitation, we propose TempTest, the first fully model-agnostic zero-shot detector that characterizes machine-generated text by modeling the conditional probability normalization distortion induced by temperature- or top-k–based decoding. TempTest quantifies local probability distribution distortion, analyzes normalization sensitivity, and formulates a theoretically grounded zero-shot hypothesis testing framework—ensuring both provable guarantees and strong interpretability. Empirically, TempTest achieves state-of-the-art performance across diverse LMs, multiple benchmark datasets, and varying text lengths. It demonstrates robustness against paraphrasing attacks and exhibits no significant bias against non-native English speakers.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Existing methods for the zero-shot detection of machine-generated text are dominated by three statistical quantities: log-likelihood, log-rank, and entropy. As language models mimic the distribution of human text ever closer, this will limit our ability to build effective detection algorithms. To combat this, we introduce a method for detecting machine-generated text that is entirely agnostic of the generating language model. This is achieved by targeting a defect in the way that decoding strategies, such as temperature or top-k sampling, normalize conditional probability measures. This method can be rigorously theoretically justified, is easily explainable, and is conceptually distinct from existing methods for detecting machine-generated text. We evaluate our detector in the white and black box settings across various language models, datasets, and passage lengths. We also study the effect of paraphrasing attacks on our detector and the extent to which it is biased against non-native speakers. In each of these settings, the performance of our test is at least comparable to that of other state-of-the-art text detectors, and in some cases, we strongly outperform these baselines.

Problem

• Detect machine-generated text without model knowledge
• Address normalization defects in decoding strategies
• Evaluate detector performance across diverse conditions

Innovation

• Detects machine text without model specifics
• Targets decoding strategy normalization defects
• Theoretically justified and explainable approach