Looking for the Inner Music: Probing LLMs' Understanding of Literary Style

2025-02-05

AI Summary

This work investigates how large language models (LLMs) distinguish author style from genre style, and the underlying representational mechanisms. We employ a multi-faceted analytical framework—including syntactic masking, attention head attribution, neuron activation tracking, controlled ablation, and style-classification fine-tuning—to systematically compare recognition pathways for these two stylistic dimensions. Results demonstrate that LLMs exhibit strong discriminative capability for both authorship and genre, yet rely on fundamentally distinct strategies: author style is primarily captured through local syntactic patterns and context-sensitive lexical choices, whereas genre style depends more heavily on global structural cues. Notably, pronoun usage and word order emerge as highly discriminative features across hierarchical model layers in both tasks. The study further reveals the critical role of fine-grained linguistic features—such as syntactic fine-tuning—in stylistic representation, providing interpretable evidence for how LLMs model stylistic variation.

Technology Category

• Natural Language Processing: Sentiment Analysis, Stylistic Analysis, and Argument Mining
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Recent work has demonstrated that language models can be trained to identify the author of much shorter literary passages than has been thought feasible for traditional stylometry. We replicate these results for authorship and extend them to a new dataset measuring novel genre. We find that LLMs are able to distinguish authorship and genre, but they do so in different ways. Some models seem to rely more on memorization, while others benefit more from training to learn author/genre characteristics. We then use three methods to probe one high-performing LLM for features that define style. These include direct syntactic ablations to input text as well as two methods that look at model internals. We find that authorial style is easier to define than genre-level style and is more impacted by minor syntactic decisions and contextual word usage. However, some traits like pronoun usage and word order prove significant for defining both kinds of literary style.

Problem

• LLMs distinguish authorship and genre
• Models use memorization or training for style
• Authorial style defined by syntax and word usage

Innovation

• LLMs distinguish authorship via training
• Syntactic ablations probe literary style
• Pronoun usage defines literary styles