🤖 AI Summary
This work investigates how large language models encode non-factual literary style—particularly authorial style—in deep prompt embeddings, beyond semantic or factual content representation. Methodologically, it leverages Transformer-based architectures and employs geometric analysis of embedding spaces, cross-text style clustering, and visualization to systematically characterize the distributional properties of short texts in high-dimensional latent space. The study reveals, for the first time, that deep embeddings of texts by the same author exhibit strong aggregation and entanglement, while those from different authors are markedly separated; moreover, the geometric structure of these embeddings stably encodes abstract stylistic features. These findings demonstrate that prompt embeddings serve not merely as compressed semantic representations but also as compact, structured encodings of stylistic information. Consequently, this work establishes a novel, interpretable, and computationally tractable paradigm for author attribution, stylometry, and related tasks.
📝 Abstract
Large language models use high-dimensional latent spaces to encode and process textual information. Much work has investigated how the conceptual content of words translates into geometrical relationships between their vector representations. Fewer studies analyze how the cumulative information of an entire prompt becomes condensed into individual embeddings under the action of transformer layers. We use literary pieces to show that information about intangible, rather than factual, aspects of the prompt are contained in deep representations. We observe that short excerpts (10 - 100 tokens) from different novels separate in the latent space independently from what next-token prediction they converge towards. Ensembles from books from the same authors are much more entangled than across authors, suggesting that embeddings encode stylistic features. This geometry of style may have applications for authorship attribution and literary analysis, but most importantly reveals the sophistication of information processing and compression accomplished by language models.