<?xml version="1.0" standalone="yes"?>
<Paper uid="W99-0406">
  <Title>Dual Use of Linguistic Resources: Evaluation of MT Systems and Language Learners</Title>
  <Section position="4" start_page="33" end_page="36" type="metho">
    <SectionTitle>
3. Using Linguistic Resources to
</SectionTitle>
    <Paragraph position="0"> Evaluate an MT System One of the objectives of our work is to support users of the embedded MT systems that our laboratory has been involved in developing. These systems were designed to be 'good enough' for filtering or relevance analysis of hardcopy, open-source text documents. 7 The ESE dataset was developed as part of an ongoing effort to expand our evaluation test suites. Here we report on a preliminary test that explored the feasibility of using sentences from the ESE dataset with their human translation into French, to evaluate one MT engine that we know is being used in the field. Eight Sentences from the ESE dataset were selected and run through the MT engine from English to French. The results of these automatic translations were then compared to the human translator's 7 Church and Hovy (1993) spelled out this notion of 'good enough' MT, and Resnik (1997) has introduced a clever method to test this.</Paragraph>
    <Paragraph position="1">  translations. Two groups of prepositions, corresponding to the two types of divergences discussed above, were of interest to us.</Paragraph>
    <Section position="1" start_page="34" end_page="34" type="sub_section">
      <SectionTitle>
3.1 Default Place Readings
</SectionTitle>
      <Paragraph position="0"> First, we were curious about how ambiguous path/place readings were handled, given that the MT engine we were working with was designed to produce only one preferred translation per input sentence, as is common for commercial MT products. We predicted that only the place reading would appear in the French MT results. We knew from discussions with MT developers that they rely heavily on hand-coded dictionaries in creating their on-line lexicons. Since English and English-French dictionaries list locational prepositions, such as those in examples 2 and 3, with only a place reading, not a path reading, it seemed most likely that only the place reading would appear in the French MT results.</Paragraph>
      <Paragraph position="1"> Another reason we expected place readings for the ambiguous phrases was that they are the direct result of the shortest path through an MT system, that is, via simple word replacements. Our predictions proved correct. Five sentences were ambiguous with both place and path readings, but all received only a place reading in the MT translations:  Test Sentences MT Output 1. He danced behind 1.11 a dans6 derriere l'6eran. the screen.</Paragraph>
      <Paragraph position="2"> 2. He carried his luggage 2. I1 a port6 son bagage dans in the airplane, l'avion.</Paragraph>
      <Paragraph position="3"> 3. He carried his luggage 3. I1 a port6 son bagage ~t inside the restaurant, l'int6fieur du restaurant. 4. He jumped on the bed. 4. I1 a saut6 sur le lit. 5. They danced in the 5. lls ont darts6 dans la  room. chambre.</Paragraph>
      <Paragraph position="4"> These results led us to predict that in our sentence 'The troops marched in the canyon', the MT engine would produce only the translation that meant the troops were marching while remaining in the canyon. This was indeed what the engine produced when we tested it.</Paragraph>
    </Section>
    <Section position="2" start_page="34" end_page="34" type="sub_section">
      <SectionTitle>
3.2 True Path Readings
</SectionTitle>
      <Paragraph position="0"> Second, we wanted to see what happened to the unambiguous path readings, given that the MT engine needed only a lexical pattern recognition to detect the English verb-preposition combination and then follow the well-documented conversion to French (Dorr, 1994). As shown in example 4, the English spatial semantics is redistributed: the manner of motion in the main verb is moved out to an adjunct in the French (en marchant), while the motion of going into the canyon is lexicalized in the main French verb and  preposition (entrer and dans).</Paragraph>
      <Paragraph position="1"> 4. E: The troops marched into the canyon. F: Les soldats sont entr6s dans la gorge en marchant.</Paragraph>
      <Paragraph position="2"> g: The troops entered in the canyon marching.  We suspected however that the unambiguous path readings might not be properly detected, given the English-French divergence with respect to directional particles and prepositions discussed above. The results are given below:</Paragraph>
    </Section>
    <Section position="3" start_page="34" end_page="35" type="sub_section">
      <SectionTitle>
Test Sentences
</SectionTitle>
      <Paragraph position="0"> 1. He carried his luggage across the street.</Paragraph>
      <Paragraph position="1"> 2. He climbed down the mountain.</Paragraph>
      <Paragraph position="2"> 3. The woman jumped out of the cake.</Paragraph>
      <Paragraph position="3"> MT Output 1. II a port6 son bagage ~t travers la rue.</Paragraph>
      <Paragraph position="4"> 2. I1 s'est 61ev6 en bas de la montagne.</Paragraph>
      <Paragraph position="5"> 3. La femme a saut6 du g~.teau, s  Our suspicions were correct; the MT engine did not correctly translate the three unambiguous path-only readings we 8 Although technically correct, this translation is the result of a &amp;quot;simple word replacement&amp;quot; strategy on the part of the MT system, and not a sophisticated translation using semantic interpretation.</Paragraph>
      <Paragraph position="6">  tested. Surprisingly, the actual MTgenerated translations failed to capture any path interpretation at all. Example 5 below shows that the MT system again produced the direct result of the shortest path through an MT system, with simple word replacements. Since the English into translated to dans, the overall result was incorrect: the translation produced the unambiguous French place-only reading.  5. E: The troops marched into the canyon. MT-F output:  Les soldats ont march6 dans la gorge. g: The troops marched in the canyon. The results of the MT experiment allow us to conclude that for 'true-path' pattern sentences, the MT system will most likely fail to output an accurate translation. Our predictions for the behavior of the MT engine on the first group of prepositions proved correct. On the second group of prepositions, we predicted accurately that the MT engine would not produce the correct translation; however, we failed to predict the specific translations that were output. The MT engine that we are working with allows users to create their own lexicon entries that supercede those of the built-in general-purpose system lexicon. Our next steps will be to test other prepositions and to examine how the lexicon entries we create will alter the translations.</Paragraph>
    </Section>
    <Section position="4" start_page="35" end_page="36" type="sub_section">
      <SectionTitle>
4. Using Linguistic Resources to
Evaluate Language Learners
</SectionTitle>
      <Paragraph position="0"> We are interested in the idea that learners can benefit from viewing parallel sentence-aligned text, as has been explored for cross-training of French speakers learning Haitian Creole (Rincher, 1986). We would expect that divergences are readily understood by language learners when presented with parallel text. Our first step, however, before exploring this possibility for teaching, has been to use the ESE dataset to evaluate second language learners to determine if they encounter the problems with spatial language that the MT system did.</Paragraph>
      <Paragraph position="1"> Fourteen intermediate-level French language learners were given the same sentences from the data set used in the MT pilot experiment and were asked to translate into French. They were told explicitly that some of the sentences might be ambiguous. They were also given a spatial expression that was ambiguous as an example and the two interpretations of that expression were explained with paraphrases.</Paragraph>
      <Paragraph position="2">  Because their level of French was not high, the college students were not always aware of the divergence in the expression of spatial paths. When faced with unambiguous path sentences (&amp;quot;true path&amp;quot; column in data table), the majority gave a simple word replacement translation, just as we had found in the MT system output. None of the students were able to correctly translate all three test sentences.</Paragraph>
      <Paragraph position="3"> In contrast to this, when translating into French the English sentences with default place-type prepositions (&amp;quot;default place&amp;quot; column in data table), a few students were able to consistently incorporate the spatial meaning of the English preposition into the French verb and properly disambiguate  the test sentences. Nonetheless, these students were not able to use this knowledge in their translations of the &amp;quot;true path&amp;quot; sentences.</Paragraph>
      <Paragraph position="4"> This pilot experiment has given us a preliminary look at learners' understanding of cross-linguistic divergences in spatial expressions.</Paragraph>
      <Paragraph position="5"> Further testing of this domain with other sentences and with more advanced students is still needed.</Paragraph>
      <Paragraph position="6"> Conclusions We have developed a test suite of spatial expressions as part of our ongoing support work evaluating the embedded MT system prototypes and the language sustainment tools being developed inhouse. The French language examples discussed above show how problematic the domain of spatial language is for both MT and for language learners.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>