Can Artificial Intelligence Write Like Borges? An Evaluation Protocol for Spanish Microfiction

2025-06-09

AI Summary

This study addresses the challenge of evaluating the literary merit of AI-generated Spanish microfiction. We propose GrAImes, the first systematic evaluation framework integrating classical literary criticism theory. GrAImes operationalizes literary assessment across four hierarchical dimensions: thematic coherence, textual clarity, interpretive depth, and aesthetic quality. It is empirically validated through dual-group annotation—by literary scholars and experienced readers—employing qualitative coding and inter-annotator agreement analysis (Cohen’s κ = 0.82). Compared to existing automated metrics, GrAImes demonstrates significantly superior discriminative power, particularly in interpretive and aesthetic dimensions. Its core contribution lies in the first structured incorporation of literary theory into AI text evaluation paradigms, establishing a novel standard for generative literary quality assessment that bridges theoretical rigor and empirical validity.

Technology Category

• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Natural Language Processing: Generation
• Machine Learning: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content

Abstract

Automated story writing has been a subject of study for over 60 years. Large language models can generate narratively consistent and linguistically coherent short fiction texts. Despite these advancements, rigorous assessment of such outputs for literary merit - especially concerning aesthetic qualities - has received scant attention. In this paper, we address the challenge of evaluating AI-generated microfictions and argue that this task requires consideration of literary criteria across various aspects of the text, such as thematic coherence, textual clarity, interpretive depth, and aesthetic quality. To facilitate this, we present GrAImes: an evaluation protocol grounded in literary theory, specifically drawing from a literary perspective, to offer an objective framework for assessing AI-generated microfiction. Furthermore, we report the results of our validation of the evaluation protocol, as answered by both literature experts and literary enthusiasts. This protocol will serve as a foundation for evaluating automatically generated microfictions and assessing their literary value.

Problem

• Evaluating AI-generated microfiction for literary merit
• Assessing aesthetic qualities in AI-written Spanish microfictions
• Developing objective literary criteria for AI story evaluation

Innovation

• GrAImes protocol for AI microfiction evaluation
• Literary theory-based objective assessment framework
• Validation by experts and enthusiasts