METRICALARGS: A Taxonomy for Studying Metrical Poetry with LLMs

2025-10-09

AI Summary

Current NLP research on poetry predominantly focuses on generation and summarization, overlooking the diagnostic value of metrical poetry—subject to strict constraints on syllables, phonemes, and prosodic rules—for probing deep linguistic understanding and rule adherence in language models. Method: We propose METRICALARGS, the first multidimensional NLP task taxonomy for metrical poetry, comprising four task categories—analysis, retrieval, generation, and support—and introduce a prosody-aware evaluation framework grounded in metrical theory. Using Telugu as a case study, we design a dedicated dataset, define linguistically informed metrics, and conduct empirical validation. Contribution/Results: This work systematically identifies critical capability bottlenecks of large language models under stringent formal constraints for the first time. It establishes a scalable, multilingual methodology for metrical analysis and advances interdisciplinary research at the intersection of computational poetics and model evaluation.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

Prior NLP work studying poetry has focused primarily on automatic poem generation and summarization. Many languages have well-studied traditions of poetic meter which enforce constraints on a poem in terms of syllable and phoneme patterns. Such advanced literary forms offer opportunities for probing deeper reasoning and language understanding in Large Language Models (LLMs) and their ability to follow strict pre-requisites and rules. In this paper, we introduce MetricalARGS, the first taxonomy of poetry-related NLP tasks designed to evaluate LLMs on metrical poetry across four dimensions: Analysis, Retrieval, Generation, and Support. We discuss how these tasks relate to existing NLP tasks, addressing questions around datasets and evaluation metrics. Taking Telugu as our example language, we illustrate how the taxonomy can be used in practice. MetricalARGS highlights the broader possibilities for understanding the capabilities and limitations of today's LLMs through the lens of metrical poetry.

Problem

• Evaluating LLMs on metrical poetry analysis, retrieval, generation, and support
• Studying LLM capabilities in following strict poetic meter constraints
• Developing taxonomy for metrical poetry tasks across multiple languages

Innovation

• Introduces MetricalARGS taxonomy for poetry analysis
• Evaluates LLMs across four poetic dimensions
• Uses Telugu poetry to demonstrate practical application