<?xml version="1.0" standalone="yes"?>
<Paper uid="P98-2139">
  <Title>Deriving Transfer Rules from Dominance-Preserving Alignments</Title>
  <Section position="7" start_page="846" end_page="846" type="concl">
    <SectionTitle>
6 Evaluation
</SectionTitle>
    <Paragraph position="0"> Real evaluation of performance of MT systems is time consuming and subjective. Nevertheless, some evaluation system is needed to insure that incremental changes are for the better, or at least, are not detrimental. We measured the success of our translation by how closely we reproduced Microsoft's English (target language) text. Our evaluation procedure computes the ratio between (a) the complement of the intersection set of words in our translation and the actual Microsoft sentence; and (b) the combined lengths of these two sentences. An exact translation gives a score of 0. If the system generates the sentence &amp;quot;A B C D E&amp;quot; and the actual sentence is &amp;quot;A B C F&amp;quot;, the score is 3/9 (the length of D E F divided by the combined lengths of A B C D E and A B C F.) The dominance-preserving version of the program produced output for 88 out of 91 test sentences. The average score for these 88 sentences was 0.29:0.21 due to incorrect word matches and 0.08 due to failure to translate because insufficient confidence levels were reached. The LCA-preserving version produced output for only 83 sentences with an average score of over 0.30: about 0.23 due to incorrect word matches and about 0.08 due to insufficient confidence levels. This crude scoring technique suggests that the dominance-preserving algorithm improved our results: more sentences were translated with higher quality. One limitation of this scoring technique is that paraphrases are penalized. An imperfect score (even .20) may signify an adequate translation.</Paragraph>
  </Section>
class="xml-element"></Paper>