<?xml version="1.0" standalone="yes"?>
<Paper uid="W97-0114">
  <Title>I I I I I</Title>
  <Section position="4" start_page="136" end_page="138" type="intro">
    <SectionTitle>
4 Evaluation
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="136" end_page="137" type="sub_section">
      <SectionTitle>
4.1 Evaluation Method
</SectionTitle>
      <Paragraph position="0"> The method to identify zero pronouns and their antecedents within Japanese and English aligned sentence pairs which was discussed in Section 3 was evaluated by automatically identifying zero pronouns and their antecedents from the functional test sentence set which is already aligned sentence by sentence. The conditions for the evaluation were as follows:  The evaluation used Japanese and English aligned sentence pairs which contain zero pronouns (554 instances; 437 sentences) in a test set designed to evaluate the performance of Japanese-to-English machine translation systems (Ikehara et al., 1994) (3718 sentence pairs).</Paragraph>
      <Paragraph position="1">  For the sentence pairs which contain zero pronouns, the syntactic and semantic structure of each Japanese sentence was created using the Japanese-to-English MT system, ALT-J/E as the Japanese analyzer, described in section 3.1, and the partial syntactic structure of each English sentence was created using Brill's tagger as the English analyzer, described in section 3.2. To avoid any effects caused by problems at the analysis step on the evaluation of the rules to identify antecedents of zero pronouns, described in section 3.3, any incorrect structures in the Japanese and English sentences were modified by hand for the evaluation. Furthermore, to take into account the effect  of the anaphora resolution of anaphoric expression in English sentences on the accuracy of the identification of the antecedents of zero pronouns, the antecedents of anaphoric expression in the English sentences were determined by hand and the accuracy with and without anaphora resolution was compared.</Paragraph>
      <Paragraph position="2">  For the sentence pairs which contain zero pronouns, antecedents of each zero pronoun within the text set were automatically identified (Section 3).</Paragraph>
      <Paragraph position="3">  To ex~m~e the effectiveness of automatically identifying antecedents of Japanese zero pronouns within English translations, the accuracy of the identified antecedents of each zero pronoun of the following three types were examined.</Paragraph>
      <Paragraph position="4">  * Zero pronouns with intrasentential antecedents (150 instances) * Zero pronouns with deictic referents which are not expressed in English translation by using passive voice expressions (157 instances) * Zero pronouns with delctic referents which are expressed in English translation (247 instances)  When rules in Section 3.3 correctly identify the accurate antecedent of a Japanese zero pronoun, with intrasentential antecedents or with deictic referent, which are expressed in the English translation, or when rules in Section 3.3 correctly identify a Japanese zero pronoun as unalignable, that is there is no deictic referents expressed in the English translation, the antecedents of zero pronoun are judged to be identified successfully.</Paragraph>
    </Section>
    <Section position="2" start_page="137" end_page="138" type="sub_section">
      <SectionTitle>
4.2 Identification Accuracy
</SectionTitle>
      <Paragraph position="0"> The accuracy of identified antecedents of zero pronouns is shown in Table 2. As shown in this table, the accuracy of identified antecedents for three types of zero pronouns using rules described in Section 3.3 is as high as 91.5% in the test using an English analyzer with anaphora resolution and 87.2% even in the test using an English analyzer without anaphora resolution. In particular, the identification of unatignable for zero pronouns with deictic referents which are not expressed in English translation is 100~ accurate in both tests. Furthermore, according to these results, the anaphora resolution in English only affects the accuracy of antecedent identification for the zero pronouns with intrasentential antecedents (89.3% with anaphora resolution, 73.3% without anaphora resolution). This result shows that even without using anaphora resolution at English analysis, this method achieves relatively high accuracy for zero pronouns with intrasententiai antecedents.</Paragraph>
      <Paragraph position="1"> The result of the detailed exam~ration of the zero pronouns whose antecedents can not be identified correctly using this method (47 instances) are summarized in Table 3. The most common cause of error is that the Japanese sentence was translated freely and there is no corresponding antecedent within the English translation (42 out of 47 instances). For these sentence pairs, the antecedents of the zero pronouns are not explicitly expressed in the human translation, but the Japanese analyzer still needs the resolution of the zero pronouns because the machine translation system which contains the Japanese analyzer can not translate freely like the human translator.</Paragraph>
      <Paragraph position="2">  So, this problem is caused by the methodological limitation of proposed method. The remaining 5 instances are caused by zero pronouns with intrasentential antecedents. In 4 out of 5 cases the antecedents of the zero pronouns were not explicitly translated in English by using the passive voice but humans can easily understand that the antecedents of the zero pronouns are in the same Sentence. This problem is also caused by the limitations of proposed method. The remaining case is caused by the fact that the verb of the zero pronoun and the verb of their antecedents could not be aligned. This is not a \]imitation of proposed method but a \]imitation of the alignment algorithm. These results shows that, for the test set, 508 out of 554 zero pronouns can have their antecedents identified using proposed method methodologically, and ~\]most all of the antecedents of zero pronouns which can be identified from the Japanese and English alig)aed sentence pairs are correctly identified by using proposed method (99.8% ; 507 out of 508 instances).</Paragraph>
      <Paragraph position="3">  the cause of the ~or &amp;quot;&amp;quot; U Intrasenten~al Deictic(explic/t) Total free translation 11 31 42 pass/ve voice 4 0 4 pre.dicate can not be aligned 1 &amp;quot; 0 i According to these results, the proposed method is effective for the automatic identification of Japanese zero pronouns and their antecedents from Japanese and English aligned sentence pairs and by using a large amount of aligned sentence pairs it is possible to identify antecedents of Japanese zero pronouns for almost all types of Japanese zero pronouns.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>