CHILL at SemEval-2025 Task 2: You Can't Just Throw Entities and Hope -- Make Your LLM to Get Them Right

2025-06-16

AI Summary

To address insufficient accuracy in named entity translation, this paper proposes an entity-aware retrieval-augmented and LLM-driven iterative correction framework. Methodologically: (1) Retrieval-augmented generation (RAG) is employed for context-sensitive entity alignment; (2) a dual-dimension self-evaluation mechanism—combining rule-based and semantic criteria—is designed, enabling the LLM to jointly assess entity translation correctness and overall translation quality; (3) multi-round self-correction is performed based on evaluation feedback, achieving end-to-end optimization of entity consistency. Experiments on SemEval-2025 Task 2 demonstrate significant improvements: +4.2 F1 score on named entities and +1.8 BLEU score, with strong robustness and generalization in domain-specific settings. This work introduces the first LLM-coordinated evaluation-correction closed loop, establishing a novel paradigm for entity-sensitive machine translation.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

In this paper, we describe our approach for the SemEval 2025 Task 2 on Entity-Aware Machine Translation (EA-MT). Our system aims to improve the accuracy of translating named entities by combining two key approaches: Retrieval Augmented Generation (RAG) and iterative self-refinement techniques using Large Language Models (LLMs). A distinctive feature of our system is its self-evaluation mechanism, where the LLM assesses its own translations based on two key criteria: the accuracy of entity translations and overall translation quality. We demonstrate how these methods work together and effectively improve entity handling while maintaining high-quality translations.

Problem

• Improving accuracy of translating named entities
• Combining RAG and iterative self-refinement techniques
• Self-evaluation mechanism for entity and translation quality

Innovation

• Combines RAG and iterative self-refinement techniques
• Uses LLM self-evaluation for translation accuracy
• Improves entity handling with high-quality translations