DITING: A Multi-Agent Evaluation Framework for Benchmarking Web Novel Translation

2025-10-10

AI Summary

Existing evaluation frameworks for web novel translation lack domain-specific adaptability. Method: This paper introduces DITING, the first comprehensive evaluation framework tailored to web novel translation, covering six dimensions—idioms, ambiguity, terminology, tense, zero-pronouns, and cultural safety—and proposes AgentEval, a novel multi-agent reasoning-based assessment method that models translation fidelity and stylistic consistency via simulated expert decision-making. Additionally, we construct MetricAlign, a benchmark dataset comprising 18K human-annotated Chinese–English sentence pairs. Contribution/Results: Experiments across 14 mainstream models demonstrate that Chinese pretrained models (e.g., DeepSeek-V3) significantly outperform English counterparts of comparable scale. AgentEval achieves substantially higher correlation with human judgments than existing automatic metrics, advancing translation evaluation from surface-level lexical matching toward deep semantic and cultural alignment.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Multiagent Systems: Adversarial Agents
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large language models (LLMs) have substantially advanced machine translation (MT), yet their effectiveness in translating web novels remains unclear. Existing benchmarks rely on surface-level metrics that fail to capture the distinctive traits of this genre. To address these gaps, we introduce DITING, the first comprehensive evaluation framework for web novel translation, assessing narrative and cultural fidelity across six dimensions: idiom translation, lexical ambiguity, terminology localization, tense consistency, zero-pronoun resolution, and cultural safety, supported by over 18K expert-annotated Chinese-English sentence pairs. We further propose AgentEval, a reasoning-driven multi-agent evaluation framework that simulates expert deliberation to assess translation quality beyond lexical overlap, achieving the highest correlation with human judgments among seven tested automatic metrics. To enable metric comparison, we develop MetricAlign, a meta-evaluation dataset of 300 sentence pairs annotated with error labels and scalar quality scores. Comprehensive evaluation of fourteen open, closed, and commercial models reveals that Chinese-trained LLMs surpass larger foreign counterparts, and that DeepSeek-V3 delivers the most faithful and stylistically coherent translations. Our work establishes a new paradigm for exploring LLM-based web novel translation and provides public resources to advance future research.

Problem

• Evaluating LLM effectiveness in translating web novels comprehensively
• Assessing narrative and cultural fidelity across six key dimensions
• Developing multi-agent framework for translation quality beyond lexical metrics

Innovation

• Multi-agent framework simulates expert translation evaluation
• Six-dimensional metric assesses narrative and cultural fidelity
• Meta-evaluation dataset enables automatic metric comparison