<?xml version="1.0" standalone="yes"?>
<Paper uid="H94-1024">
  <Title>Evaluation in the ARPA Machine Translation Program:</Title>
  <Section position="2" start_page="0" end_page="0" type="intro">
    <SectionTitle>
1. INTRODUCTION.
</SectionTitle>
    <Paragraph position="0"> Evaluation of Machine Translation (MT) has proven to be a particularly difficult challenge over the course of its history. As has been noted elsewhere (White et al., 1993), assessment of how well an expression in one language is conveyed in another is loaded with subjective judgments, even when the expressions are translated by professional translators. Among these judgments are the extent to which the information was conveyed accurately, and the extent to which the information conveyed was fluently expressed in the target language. The inherent subjectivity has been noted, and attempts have been made in MT evaluation to use such judgments to best qualitative advantage (e.g., van Slype 1979). The means of capturing judgments into quantifiably useful comparisons among systems have led to legitimate constraints on the range of the evaluation, such as to the scope of the intended end-use (Church and Hovy 1991), or to the effectiveness of the linguistic model (Jordan et al. 1992, Nomura 1992, Gamback et al. 1991).</Paragraph>
    <Paragraph position="1"> The ARPA MT Initiative encompasses radically different approaches, potential end-uses, and languages.</Paragraph>
    <Paragraph position="2"> Consequently, the evaluation methodologies developed for it must capture quantifiable judgments from subjectivity, while being relatively unconstrained otherwise. This paper presents the 1993 methodologies, and the results of the 1993 MT evaluation. We further discuss the preliminary status of an evaluation now underway that greatly increases the participation of the entire MT community, while refining the sensitivity and portability of the evaluation techniques.</Paragraph>
  </Section>
class="xml-element"></Paper>