<?xml version="1.0" standalone="yes"?>
<Paper uid="W01-1405">
  <Title>Stochastic Modelling: From Pattern Classification to Language Translation</Title>
  <Section position="5" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
3 Experimental Results
</SectionTitle>
    <Paragraph position="0"> Whereas stochastic modelling is widely used in speech recognition, there are so far only a few research groups that apply stochastic modelling to language translation (Berger et al. 1994; Brown et al. 1993; Knight 1999). The presentation here is based on work carried out in the framework of the EUTRANS project (Casacuberta et al. 2001) and the VERBMOBIL project (Wahlster 2000).</Paragraph>
    <Paragraph position="1"> We will consider the experimental results obtained in the VERBMOBIL project. The goal of the VERBMOBIL project is the translation of spoken dialogues in the domains of appointment scheduling and travel planning. The languages are German and English. Whereas during the progress of the project many offline tests were carried out for the optimization and tuning of the statistical approach, the most important evaluation was the final evaluation of the VERBMOBIL prototype in spring 2000. This end-to-end evaluation of the VERBMOBIL system was performed at the University of Hamburg (Tessiore et al. 2000). In each session of this evaluation, two native speakers conducted a dialogue. The speakers did not have any direct contact and could only interact by speaking and listening to the VERBMOBIL system.</Paragraph>
    <Paragraph position="2"> In addition to the statistical approach, three other translation approaches had been integrated into the VERBMOBIL prototype system (Wahlster 2000): + a classical transfer approach, which is based on a manually designed analysis grammar, a set of transfer rules, and a generation grammar, + a dialogue act based approach, which amounts to a sort of slot filling by classifying each sentence into one out of a small number of possible sentence patterns and filling in the slot values, + an example based approach, where a sort of nearest neighbour concept is applied to the set of bilingual training sentence pairs after suitable preprocessing.</Paragraph>
    <Paragraph position="3"> In the final end-to-end evaluation, human evaluators judged the translation quality for each of the four translation results using the following criterion: Is the sentence approximatively correct: yes/no? The evaluators were asked to pay particular attention to the semantic information (e.g. date and place of meeting, participants etc.) contained in the translation. A missing translation as it may happen for the transfer approach or other approaches was counted as wrong translation.</Paragraph>
    <Paragraph position="4"> The evaluation was based on 5069 dialogue turns for the translation from German to English and on 4136 dialogue turns for the translation from  English to German. The speech recognizers used had a word error rate of about 25%. The overall sentence error rates, i.e. resulting from recognition and translation, are summarized in Table 1. As we can see, the error rates for the statistical approach are smaller by a factor of about 2 in comparison with the other approaches. In agreement with other evaluation experiments, these experiments show that the statistical modelling approach may be comparable to or better than the conventional rule-based approach. In particular, the statistical approach seems to have the advantage if robustness is important, e.g. when the input string is not grammatically correct or when it is corrupted by recognition errors.</Paragraph>
  </Section>
class="xml-element"></Paper>