<?xml version="1.0" standalone="yes"?>
<Paper uid="H94-1024">
  <Title>Evaluation in the ARPA Machine Translation Program:</Title>
  <Section position="10" start_page="136" end_page="137" type="concl">
    <SectionTitle>
1994 EVALUATION IN PROGRESS
</SectionTitle>
    <Paragraph position="0"> The 1994 Evaluation presently underway focuses on core FAMT technology. Its scope has been broadened to increase sensitivity and portability. In keeping with the ARPA MT Initiative goal to foster development of FAMT, input will move away from HAMT and include a larger proportion of FAMT. To better measure the expanded lexical capabilities of the systems under development, half of the test passages will be general news articles. The Winter 1994 evaluation alone will  generate 25,000 data points to manage human subjectivity. This increase in data points has been accomplished by successfully porting the methodology to evaluation of 14 production systems in addition to the three ARPA research systems. To maximize the randomness of passage assignment in the evaluation matrix, the Latin square has been replaced with a matrix ordered by a random number generator. The methodology has been simplified to optimize the elicitation of intuitive judgments. For example, the fluency component which formerly measured only wellformedoess has been modified to recognize the influence of contextual meaning.</Paragraph>
    <Paragraph position="1"> The broadened scope of the 1994 Evaluation offers benefits, for the evaluation of the core technology for the profoundly different systems of the ARPA MT Initiative. It also contributes to the advancement of the MT community as a whole through providing a consistent portable suite of evaluation methodologies.</Paragraph>
  </Section>
class="xml-element"></Paper>