A Comprehensive Analysis of Large Language Model Outputs: Similarity, Diversity, and Bias

2025-05-14

AI Summary

This work systematically investigates the similarity, diversity, and ethical compliance of large language model (LLM) outputs. We introduce the first cross-model, multi-dimensional quantitative evaluation framework, built upon 3 million text generations from 12 state-of-the-art models across 5,000 diverse prompts. Our framework integrates semantic similarity (BERTScore, SBERT), lexical diversity (TF-IDF, entropy), and ethical bias metrics (gender co-occurrence, stereotype scores), augmented by clustering analysis to characterize stylistic specificity. Empirically, we reveal a pervasive homogenization tendency across LLMs—strongly correlated with architectural design and training strategies: GPT-4 exhibits the highest output diversity and uniqueness; WizardLM-2-8x22B shows the strongest convergence; Llama 3 and Mistral demonstrate high stylistic similarity; several open-source models significantly improve gender balance. The study delivers a benchmark dataset and reproducible methodology for assessing LLM trustworthiness.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large Language Models (LLMs) represent a major step toward artificial general intelligence, significantly advancing our ability to interact with technology. While LLMs perform well on Natural Language Processing tasks -- such as translation, generation, code writing, and summarization -- questions remain about their output similarity, variability, and ethical implications. For instance, how similar are texts generated by the same model? How does this compare across different models? And which models best uphold ethical standards? To investigate, we used 5{,}000 prompts spanning diverse tasks like generation, explanation, and rewriting. This resulted in approximately 3 million texts from 12 LLMs, including proprietary and open-source systems from OpenAI, Google, Microsoft, Meta, and Mistral. Key findings include: (1) outputs from the same LLM are more similar to each other than to human-written texts; (2) models like WizardLM-2-8x22b generate highly similar outputs, while GPT-4 produces more varied responses; (3) LLM writing styles differ significantly, with Llama 3 and Mistral showing higher similarity, and GPT-4 standing out for distinctiveness; (4) differences in vocabulary and tone underscore the linguistic uniqueness of LLM-generated content; (5) some LLMs demonstrate greater gender balance and reduced bias. These results offer new insights into the behavior and diversity of LLM outputs, helping guide future development and ethical evaluation.

Problem

• Analyzing similarity and diversity in LLM-generated texts
• Comparing ethical standards across different LLM models
• Investigating linguistic uniqueness and bias in LLM outputs

Innovation

• Analyzed 3 million texts from 12 diverse LLMs
• Compared output similarity and variability across models
• Assessed ethical standards and linguistic uniqueness