<?xml version="1.0" standalone="yes"?>
<Paper uid="P06-2049">
  <Title>Sydney, July 2006. c(c)2006 Association for Computational Linguistics Transformation-based Interpretation of Implicit Parallel Structures: Reconstructing the meaning of vice versa and similar linguistic operators</Title>
  <Section position="9" start_page="382" end_page="383" type="evalu">
    <SectionTitle>
4 Evaluation
</SectionTitle>
    <Paragraph position="0"> We conducted an evaluation of the parallel structure building algorithm on a sample of sentences from Europarl (Koehn, 2002), a parallel corpus of professionally translated proceedings of the European Parliament aligned at the document and sentence level. At this point, we were able to conduct only manual evaluation. This is mainly due to the fact that we did not have access to a wide-coverage semantic dependency parser for English and German.4 In this section, we present our corpus sample and the evaluation results.</Paragraph>
    <Paragraph position="1"> Evaluation sample To build the evaluation sample, we used sentence- and word-tokenized English German part of Europarl. Using regular expressions, we extracted sentences with the following patterns: (i) for English, phrases the other way a*round or vice versa (ii) for German: (ii-1) the word umgekehrt preceded by a sequence of und ( and ), oder ( or ), sondern ( but ), aber ( but ) or comma, optional one or two tokens and optional nicht ( not ), (ii-2) the word umgekehrt preceded by a sequence gilt ( holds ) and one or two optional tokens, (ii-3): the word anders(he)*rum.</Paragraph>
    <Paragraph position="2"> We obtained 137 sentences.</Paragraph>
    <Paragraph position="3"> Next, given the present limitation of our algorithm (see Section 3.3), we manually excluded those whose interpretation involved the preceding sentence or paragraph,5 as well as those in which the interpretation was explicitly spelled out. There were 27 such instances. Our nal evaluation sample consisted of 110 sentences: 82 sentences in English German pairs and 28 German-only.6  of the German sentences containing the word umgekehrt may contain phrases other than the other way round or vice versa. Depending on context, phrases such as conversely, in or the reverse, the opposite, on the contrary may be used. Here, we targeted only the other way round and vice versa phrases. If the German translation contained the word umgekehrt, andthe English source one of the alternatives to our target, in the evaluation we included only the German sentence.</Paragraph>
    <Paragraph position="4"> Category No. of instances  Distribution of categories We manually categorized the structures in our sample and marked the elements of the dependency structures that participate in the transformation. Table 1. presents the distribution of structure categories. We explicitly included counts for alternative interpretations. For example Arg/Mod means that either the Argument or Modifier transformation can be applied with the same effect, as in the sentence External policy has become internal policy, and vice versa : either the words external and internal may be swapped (Modifier), or the whole NPs external policy and internal policy (Argument). Lex means that none of the patterns was applicable and a lexical paraphrase (such as use of an antonym) needed to be performed in order to reconstruct the underlying semantics (i.e. no parallel structure was involved). Other means that there was a parallel structure involved, however, none of our patterns covered the intended transformation.</Paragraph>
    <Paragraph position="5"> Evaluation results The evaluation results are presented in Tables 2. and 3. Table 2. shows an overview of the results. The interpretation of the result categories is as follows: Correct: the algorithm returned the intended reading as a unique interpretation (this includes correct identication of lexical paraphrases (the Lex category in Table 1.); Ambig.: multiple results were returned with the intended reading among them; Wrong: the algorithm returned a wrong result (if multiple results, then the intended one was not included); Failed: the algorithm failed to recognize a parallel structure where one existed because no known pattern matched.</Paragraph>
    <Paragraph position="6"> Table 3. shows within-category results. Here, Correct result for Other means that the algorithm correctly identi ed 8 cases to which no current pattern applied. The two Wrong results for Other  mean that a pattern was identi ed, however, this pattern was not the intended one. In two cases (false-negatives), the algorithm failed to identify a pattern even though it fell into one of the known categories (Argument and Prop).</Paragraph>
    <Paragraph position="7"> Discussion The most frequently occurring pattern in our sample is Argument. This is often a plausible reading. However, in 3 of the 4 false-positives (Wrong results), the resolved incorrect structure was Arg. If we were to take Arg as baseline, aside from missing the other categories (altogether 12 instances), we would obtain the nal result of 63 Correct (as opposed to 96; after collapsing the Correct and Ambig. categories) and 15 (as opposed to 4) Wrong results.</Paragraph>
    <Paragraph position="8"> Let us take a closer look at the false-negative cases and the missed patterns. Two missed known categories involved multiple arguments of the main head: a modal modi er (modal verb) and an additive particles ( also ) in one case, and in the other, rephrasing after transformation. To improve performance on cases such as the former, we could incorporate an exclusion list of dependents that the transformation should disregard.</Paragraph>
    <Paragraph position="9"> Among the patterns currently unknown to the algorithm, we found four types (one instance of each in the sample) that we can anticipate as frequently recurring: aim and recipient constructs involving a head and its Aim- and Bene ciarydependent respectively, a temporal-sequence in which the order of the sequence elements is reversed, and a comparative structure with swapped relata. The remaining 6 structures require a more involved procedure: either the target dependent is deeply embedded or paraphrasing and/or morphological transformation of the lexemes is required.</Paragraph>
  </Section>
class="xml-element"></Paper>