Efficient Topic Model Estimation under Heavy-Tailed Document Lengths

2026-07-27

AI Summary

This work addresses the limitations of traditional topic models when applied to text corpora characterized by heavy-tailed document lengths and power-law word frequency distributions. Building on the assumption that document lengths follow a regularly varying distribution, the authors uncover a hierarchical power-law structure in word frequencies at both document and topic levels within Latent Dirichlet Allocation (LDA). They propose a novel tensor decomposition algorithm based on extreme-value normalized moments, which—unlike conventional approaches—does not require corpus-specific preprocessing and exhibits enhanced robustness to data perturbations. The method integrates extreme value theory with power-law characteristics into topic modeling for the first time. Experimental validation on the Twenty Newsgroups corpus demonstrates its effectiveness, thereby advancing the intersection of machine learning and multivariate extreme value analysis.

Abstract

Early inquiries into the statistical properties of natural language found that words tend to occur with power-law frequencies. This observation, closely associated with Zipf's law, has spurred many investigations into why this power-law pattern emerges with such regularity. Rarely, however, has this property of text been leveraged in statistical inference. In this paper, we demonstrate that the Latent Dirichlet Allocation (LDA) model can accommodate power-law word frequencies. In particular, when the document length distribution is regularly varying, the word frequency distribution admits a hierarchy of power laws across documents and topics. We further leverage this finding to develop an efficient tensor decomposition algorithm for estimating the topic matrix via the moments of normalized extreme word frequencies. Applying our algorithm to the twenty newsgroups corpus reveals that the extreme-value methodology exhibits robustness to certain choices made in the pre-processing of the data. This work furthers the recent interest in adapting machine learning methods to the study of multivariate extremes.

Problem

• topic model
• heavy-tailed document lengths
• power-law
• Latent Dirichlet Allocation
• extreme-value

Innovation

• Latent Dirichlet Allocation
• power-law word frequencies
• tensor decomposition
• extreme-value methodology
• heavy-tailed document lengths