🤖 AI Summary
This study addresses the lack of decision-oriented evaluation methodologies in current machine translation quality estimation (QE) systems. It introduces receiver operating characteristic (ROC) analysis into QE evaluation for the first time, complementing and validating against conventional metrics. Experimental results demonstrate that ROC analysis not only aligns consistently with existing evaluation outcomes but also yields actionable performance insights. By providing a clearer understanding of trade-offs between true positive and false positive rates across varying decision thresholds, this approach significantly enhances the practical utility of QE assessment and offers robust guidance for deployment decisions in real-world applications.
📝 Abstract
The increasing use of automated translation quality estimation (QE) systems calls for practical, decision-oriented methods for evaluating their performance. We propose that Receiver Operating Characteristic (ROC) analysis is a useful approach for this purpose. Our study shows that ROC analysis not only produces results consistent with currently prevalent methods, but also offers several important advantages, including actionable performance insights that support business decision-making.