<?xml version="1.0" standalone="yes"?>
<Paper uid="N06-2029">
  <Title>Exploiting Variant Corpora for Machine Translation</Title>
  <Section position="5" start_page="114" end_page="115" type="evalu">
    <SectionTitle>
4 Experiments
</SectionTitle>
    <Paragraph position="0"> The effectiveness of the proposed method is verified for the CE translation task (500 sentences) of last year's IWSLT evaluation campaign. For the experiments, we used the four statistical (SMT) and three example-based (EBMT) MT engines described in detail in (Paul et al., 2005).</Paragraph>
    <Paragraph position="1"> For evaluation, we used the BLEU metrics, which calculates the geometric mean of n-gram precision for the MT outputs found in reference translations (Papineni et al., 2002). Higher BLEU scores indicate better translations.</Paragraph>
    <Section position="1" start_page="114" end_page="114" type="sub_section">
      <SectionTitle>
4.1 Performance of Element MT Engines
</SectionTitle>
      <Paragraph position="0"> Table 2 summarizes the results of all element MT engines trained on the BTECO and BTECV corpora.</Paragraph>
      <Paragraph position="1"> The result show that the SMT engines outperform  the EBMT engines whereby the best performing system is marked with bold-face.</Paragraph>
      <Paragraph position="2"> However, depending on the variant corpus used to train the MT engines, quite different system performances are achieved. Most of the element MT engines perform better when trained on the smaller BTECV corpus indicating that the given test set is not covered well by the BTECO corpus.</Paragraph>
    </Section>
    <Section position="2" start_page="114" end_page="115" type="sub_section">
      <SectionTitle>
4.2 Effects of Hypothesis Selection
</SectionTitle>
      <Paragraph position="0"> The performance of the hypothesis selection method (SEL) is summarized in Table 3 whereby the obtained gain relative to the best element MT engine is given in parentheses. In addition, we performed an &amp;quot;oracle&amp;quot; translation experiment in order to investigate in an upper boundary for the method. Each input sentence was translated by all element MT engines and the translation hypothesis with the lowest word error rate2 relative to the reference translations was output as the translation, i.e., the ORACLE system simulates an optimal selection method according to an objective evaluation criterion.</Paragraph>
      <Paragraph position="1">  The results show that the selection method is effective for both variant corpora whereby a larger gain is achieved for BTECV . However, the ORACLE results indicate that the method fails to tap the full potential of the element MT engines.</Paragraph>
      <Paragraph position="2"> In addition, we trained the statistical models of the hypothesis selection method on the corpus obtained 2The word error rate (WER) is an objective evaluation measures that, in contrast to BLEU, can be applied on sentencelevel. It penalizes edit operations for the translation output against reference translations.</Paragraph>
      <Paragraph position="3">  by merging all variant corpora (BTECO[?]V ). Despite the larger amount of training data, the BLEU score decreases drastically which shows that an increase in training data not necessarily leads to improved translation quality. Moreover, the ORACLE selection applied to all translation hypotheses based on the BTECO as well as the BTECV corpus indicates that both variants can contribute significantly in order to improve the overall system performance.</Paragraph>
    </Section>
    <Section position="3" start_page="115" end_page="115" type="sub_section">
      <SectionTitle>
4.3 Effects of Variant Selection
</SectionTitle>
      <Paragraph position="0"> The effects of combining selected variant hypotheses by testing whether significant differences in statistical scores were obtained are summarized in Table 4. The variant selection method is applied to the translation outputs of each element MT engine (MTOj k MTVj ) as well as the selected translation hypotheses (MTOSEL k MTVSEL). The gain of the proposed variant selection method relative the best element MT output based on a single variant corpus is given in parentheses.</Paragraph>
      <Paragraph position="1">  The results show that the variant selection method is effective for all element MT engines. The highest BLEU score is achieved for MTOSEL bardbl MTVSEL gaining 4.2% in BLEU score. Moreover, the proposed method outperforms the hypothesis selection method based on the merged corpus BTECO[?]V by 11.2% in BLEU score.</Paragraph>
      <Paragraph position="2"> A comparison of the proposed method with the best performing system (C-STAR data track, BLEU=0.5279) of the IWSLT 2005 workshop showed that our system outperforms the top-ranked system gaining 4.8% in BLEU score.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>