🤖 AI Summary
This work addresses the suboptimal performance of default top-layer embeddings in BERTopic. We systematically investigate how intermediate-layer Transformer representations affect topic quality. Specifically, we evaluate 18 intermediate-layer embedding configurations across three heterogeneous text corpora, quantitatively assessing topic coherence and diversity, and analyzing how stopword removal strategies interact dynamically with embedding layer selection. Our key contributions are: (1) first empirical demonstration that intermediate-layer embeddings consistently outperform the default top-layer embeddings; (2) discovery that stopword impact is highly layer-dependent; and (3) identification of optimal configurations that significantly surpass the BERTopic baseline—achieving up to a 12.7% improvement in topic coherence. These findings establish a reproducible, embedding-level optimization paradigm for interpretable and robust topic modeling.
📝 Abstract
BERTopic is a topic modeling algorithm that leverages transformer-based embeddings to create dense clusters, enabling the estimation of topic structures and the extraction of valuable insights from a corpus of documents. This approach allows users to efficiently process large-scale text data and gain meaningful insights into its structure. While BERTopic is a powerful tool, embedding preparation can vary, including extracting representations from intermediate model layers and applying transformations to these embeddings. In this study, we evaluate 18 different embedding representations and present findings based on experiments conducted on three diverse datasets. To assess the algorithm's performance, we report topic coherence and topic diversity metrics across all experiments. Our results demonstrate that, for each dataset, it is possible to find an embedding configuration that performs better than the default setting of BERTopic. Additionally, we investigate the influence of stop words on different embedding configurations.