Abstract

The goal of translation, be it by human or by machine, is, given some textin a source language, to produce text in a target language that simultaneously 1) preserves the meaning of the source text and 2) achieves natural expression in the target language. However, researchers in the machine translation community usually assess translations using a single score in-tended to capture semantic accuracy and the naturalness of the output simultaneously. In this talk, I build on Michaeli and Blau’s the seminal work on distortion-perception theory to propose a general notion of accuracy and naturalness, and show that such single-score summaries do not and cannot give the complete picture of a translation system’s true performance. I also demonstrate the accuracy-naturalness tradeoff by evaluating the performance of the state-of-the-art translation systems on a popular benchmark. These findings help explain well-known empirical phenomena, such as the observation that optimizing translation systems for a specific accuracy metric initially improves the system’s naturalness, while “overfitting” the system to the metric can significantly degrade its naturalness. Finally, I propose a new statistical distance in the spirit of integral probability metrics to measure naturalness in practice and discuss some of its properties.