Large Language Models for Classical Chinese Poetry Translation: Benchmarking, Evaluating, and Improving

2024-08-19

AI Summary

Classical Chinese poetry translation faces a fundamental trade-off between cultural fidelity and poetic expressiveness. Method: This paper introduces PoetMT—the first fine-grained evaluation benchmark for classical Chinese poetry translation—alongside a GPT-4–driven semantic-aesthetic joint metric and a Retrieval-Augmented Translation (RAT) framework that dynamically injects domain-specific knowledge via RAG and multi-stage prompt optimization. Contribution/Results: RAT achieves state-of-the-art performance across automatic metrics (BLEU, COMET, BLEURT) and human evaluation. Notably, human evaluators report a 32% improvement in poetic accuracy, marking the first demonstration that large language models can simultaneously preserve cultural authenticity and literary artistry in classical Chinese poetry translation.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Different from the traditional translation tasks, classical Chinese poetry translation requires both adequacy and fluency in translating culturally and historically significant content and linguistic poetic elegance. Large language models (LLMs) with impressive multilingual capabilities may bring a ray of hope to achieve this extreme translation demand. This paper first introduces a suitable benchmark (PoetMT) where each Chinese poetry has a recognized elegant translation. Meanwhile, we propose a new metric based on GPT-4 to evaluate the extent to which current LLMs can meet these demands. Our empirical evaluation reveals that the existing LLMs fall short in the challenging task. Hence, we propose a Retrieval-Augmented Machine Translation (RAT) method which incorporates knowledge related to classical poetry for advancing the translation of Chinese Poetry in LLMs. Experimental results show that RAT consistently outperforms all comparison methods regarding wildly used BLEU, COMET, BLEURT, our proposed metric, and human evaluation.

Problem

• Large Language Models
• Classical Chinese Poetry Translation
• Accuracy and Poetic Retention

Innovation

• Retrieval-Augmented Machine Translation (RAT)
• Classical Chinese Poetry Translation
• Preservation of Poetic Style