Local Brushstroke Quality Assessment via Vision-Language Feedback

2026-07-15

AI Summary

This study presents the first systematic evaluation of multimodal large language models (MLLMs) in fine-grained assessment of Chinese calligraphy stroke quality. By constructing a five-point rating framework based on before-and-after image pairs and leveraging expert annotations, the authors quantify the absolute scoring accuracy and ranking correlation (Kendall’s τ) of state-of-the-art models including GPT-4o, Claude Sonnet 4, and Gemini 2.5 Flash. Results show that GPT-4o achieves the best performance (MAE = 0.885), yet none of the models attain ranking consistency comparable to human experts. Incorporating retrieval-augmented generation (RAG) improves ranking correlation at the cost of absolute scoring accuracy, revealing a trade-off between these two objectives. Furthermore, the analysis uncovers systematic evaluation biases in the models, offering new insights for intelligent calligraphy instruction.

Abstract

This paper investigates whether multimodal LLMs can evaluate local brushstroke quality in calligraphy and generate educationally useful natural language feedback. We construct an evaluation framework in which three multimodal LLMs (GPT-4o, Claude Sonnet 4, and Gemini 2.5 Flash) assess before-after image pairs of calligraphic works using a five-point ordinal scale, and compare their outputs against scores assigned by three expert calligraphers. We additionally examine a Retrieval-Augmented Generation (RAG) variant of Claude as a preliminary condition. Results show that all models achieve useful levels of absolute score accuracy (MAE), with GPT-4o performing best (MAE = 0.885). However, none of the models produce statistically significant overall rank correlations with human experts (Kendall's tau). Vocabulary analysis of generated rationales reveals characteristic evaluative biases in each model, and RAG is shown to improve rank correlation while worsening absolute accuracy, constituting an important negative result for text-based rule injection.

Problem

• brushstroke quality assessment
• multimodal LLMs
• calligraphy evaluation
• vision-language feedback
• expert alignment

Innovation

• multimodal LLMs
• brushstroke quality assessment
• vision-language feedback
• Retrieval-Augmented Generation
• calligraphy education