The dynamics of meaning through time: Assessment of Large Language Models

2025-01-09

AI Summary

This study investigates large language models’ (LLMs) capacity to comprehend historical semantic evolution of concepts. To address the lack of rigorous evaluation for temporal semantics, we propose the first multidimensional benchmark framework specifically designed for semantic temporal dynamics. Our method introduces a cross-temporal term explanation task, integrating automated metrics—perplexity, generation length, and consistency—with double-blind expert subjective evaluation, augmented by temporally grounded prompt engineering to enhance time sensitivity. Results reveal systematic generational disparities: GPT-4 and Claude exhibit significantly greater long-term semantic stability than Llama and early ChatGPT. Critically, all models manifest a pronounced “presentism” bias, with average accuracy dropping by 37% on pre-industrial terminology. These findings expose structural limitations in LLMs’ historical contextualization capabilities and establish a reproducible benchmark and methodological paradigm for temporally aware language understanding.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Understanding how large language models (LLMs) grasp the historical context of concepts and their semantic evolution is essential in advancing artificial intelligence and linguistic studies. This study aims to evaluate the capabilities of various LLMs in capturing temporal dynamics of meaning, specifically how they interpret terms across different time periods. We analyze a diverse set of terms from multiple domains, using tailored prompts and measuring responses through both objective metrics (e.g., perplexity and word count) and subjective human expert evaluations. Our comparative analysis includes prominent models like ChatGPT, GPT-4, Claude, Bard, Gemini, and Llama. Findings reveal marked differences in each model's handling of historical context and semantic shifts, highlighting both strengths and limitations in temporal semantic understanding. These insights offer a foundation for refining LLMs to better address the evolving nature of language, with implications for historical text analysis, AI design, and applications in digital humanities.

Problem

• Large Language Models
• Historical Concept Understanding
• Semantic Change Over Time

Innovation

• Large Language Models
• Semantic Change
• Historical Context Understanding