From Small to Large Language Models: Revisiting the Federalist Papers

2025-02-25

AI Summary

This study addresses author attribution in *The Federalist Papers* by evaluating the efficacy of off-the-shelf general-purpose language models—including small-scale statistical models and large language models (LLMs)—for stylometric analysis without fine-tuning. Methodologically, it systematically compares Bayesian topic embeddings (based on function words), LDA topic modeling, word/phrase embeddings, and zero-shot LLM embeddings. Results show that function-word-driven topic embeddings substantially outperform raw LLM embeddings; dimensional expansion does not consistently improve performance, challenging the assumed advantage of LLMs in interpretable tasks; and Bayesian modeling combined with topic embeddings achieves higher generalization accuracy. The core contribution is demonstrating that compact statistical models can match or exceed LLMs in author attribution while offering superior interpretability, robustness, and computational efficiency—establishing a novel unsupervised stylistic analysis paradigm for digital humanities.

Technology Category

Natural Language Processing: Sentiment Analysis, Stylistic Analysis, and Argument MiningMachine Learning: Large Multimodal Models (LMMs)Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

Web Mining and Content Analysis: Large pretrained models with web dataGraph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphsSemantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

For a long time, the authorship of the Federalist Papers had been a subject of inquiry and debate, not only by linguists and historians but also by statisticians. In what was arguably the first Bayesian case study, Mosteller and Wallace (1963) provided the first statistical evidence for attributing all disputed papers to Madison. Our paper revisits this historical dataset but from a lens of modern language models, both small and large. We review some of the more popular Large Language Model (LLM) tools and examine them from a statistical point of view in the context of text classification. We investigate whether, without any attempt to fine-tune, the general embedding constructs can be useful for stylometry and attribution. We explain differences between various word/phrase embeddings and discuss how to aggregate them in a document. Contrary to our expectations, we exemplify that dimension expansion with word embeddings may not always be beneficial for attribution relative to dimension reduction with topic embeddings. Our experiments demonstrate that default LLM embeddings (even after manual fine-tuning) may not consistently improve authorship attribution accuracy. Instead, Bayesian analysis with topic embeddings trained on ``function words"yields superior out-of-sample classification performance. This suggests that traditional (small) statistical language models, with their interpretability and solid theoretical foundation, can offer significant advantages in authorship attribution tasks. The code used in this analysis is available at github.com/sowonjeong/slm-to-llm

Problem

• Revisits authorship attribution of Federalist Papers using modern language models.
• Examines effectiveness of LLM embeddings in text classification without fine-tuning.
• Compares traditional statistical models with LLMs for authorship attribution accuracy.

Innovation

• Uses modern language models for text classification.
• Compares word and topic embeddings for authorship attribution.
• Demonstrates Bayesian analysis outperforms default LLM embeddings.