<?xml version="1.0" standalone="yes"?>
<Paper uid="W03-0307">
  <Title>Phrase-based Evaluation of Word-to-Word Alignments</Title>
  <Section position="1" start_page="0" end_page="3" type="abstr">
    <SectionTitle>
Abstract
</SectionTitle>
    <Paragraph position="0"> We evaluate the English--French word alignment data of the shared tasks from a phrase alignment perspective. We discuss peculiarities of the submitted data and the test data. We show that phrase-based evaluation is closely related to word-based evaluation. We show examples of phrases which are easy to align and also phrases which are difficult to align.</Paragraph>
    <Paragraph position="1">  Introduction We describe a phrase-based evaluation of the 16 English-French alignment submissions for the shared task on Parallel Texts. The task was to indicate which word token in an English alignment sample corresponds to which word token in the French alignment sample. Two types of submission were permitted: for restricted submissions were allowed a &amp;quot;sentence&amp;quot; aligned segment of the Canadian Hansards to train the systems while unrestricted submission would be allowed to use additional resources. The performance of the systems was compared for a set of 447 English--French hand-aligned test samples which were also taken from the Canadian Hansards.</Paragraph>
    <Paragraph position="2"> Five institutes participated in the English--French alignment task, submitting a total of 16 sets of alignment data. To evaluate the submitted data, we extracted bilingual phrase dictionaries from the word-alignment data. The extracted dictionaries of the submitted data were compared with the extracted dictionary of the test data.</Paragraph>
    <Paragraph position="3"> We first discuss word-to-word and phrase-to-phrase alignment format. We present two different methods for extracting bilingual dictionaries from the word alignment data: a minimal dictionary contains the least number of unambiguous phrase-to-phrase translations while an exhaustive dictionary contains all possible unambiguous translations. We examine the test data (i.e. the &amp;quot;golden standard&amp;quot;) and the submitted alignment data. We discuss their peculiarities and give examples of phrases easy and difficult to align.</Paragraph>
    <Section position="1" start_page="0" end_page="1" type="sub_section">
      <SectionTitle>
Types of Alignment
</SectionTitle>
      <Paragraph position="0"> The test set consists of 447 alignment samples from the Canadian Hansards which were pre-tokenized. A threetuple containing the alignment number, an English word offset and a French word offset would indicate an exact word-to-word translation  . The submitted data was supposed to comply with this word-to-word alignment format. In example 1 the English sentence has 15 tokens while the French sentence has 16 tokens. Example 1 shows the word-to-word alignment data of sample 91 for submission 12 and a plot of the data.</Paragraph>
      <Paragraph position="1"> Example 1: Alignment sample 91: English (vertical): i was not asking for a detailed explanation as to what he was doing .</Paragraph>
      <Paragraph position="2"> French (horizontal): je ne lui ai pas demande de me fournir de telles explications sur ces activites .</Paragraph>
      <Paragraph position="3">  There was also an optional slot to indicate whether this alignment would be [S]ure or [P]robable. We ignore this information in our evaluation.</Paragraph>
    </Section>
    <Section position="2" start_page="1" end_page="3" type="sub_section">
      <SectionTitle>
2.1 Word-to-word alignment
</SectionTitle>
      <Paragraph position="0"> There are two underlying assumptions in word-to-word alignment: (i) each word token on the English side can have any number of word correspondences -- including zero -- on the French side and vice versa. Word alignments may have crossing and ambiguous branches. For instance in example 1, the French word &amp;quot;me&amp;quot; on position 8 has the translations &amp;quot;was&amp;quot;, and &amp;quot;he&amp;quot;, while &amp;quot;ai&amp;quot; has no connection to the English side.</Paragraph>
      <Paragraph position="1"> (ii) words (English or French) for which no alignments are given in the submitted data are assigned a null-alignment.</Paragraph>
      <Paragraph position="2"> Example 1 has 22 word alignment points, where the evaluators inserted 7 null-alignments. In some cases (i.e. submission 11) this insertion accounts for almost 50% of the alignment data. In figure 1, &amp;quot;null-alignment&amp;quot; plots the union of the submitted alignment data and the inserted null-alignments. Null-alignments were not added to submission 16 as it provides alignment information for every word. The last data point on the x-axis (i.e. 17) (ii) an English phrase may only be unambiguously linked to exactly one French phrase and vice versa.</Paragraph>
      <Paragraph position="3"> Phrase-to-phrase alignments can be nested. For instance, the shorter English--French phrase translation 9-9 &lt;-&gt; 11-11 is included in the longer phrase translation 5-11 &lt;-&gt; 9-12: 5-11 9-12: for a detailed explanation as to what &lt;-&gt; fournir de telles explications 9-9 11-11: as &lt;-&gt; telles In this way structural information can be stored. On the other hand, we do not allow ambiguous phrase alignments as e.g.: 8-8 12-12 explanation &lt;-&gt; explications 11-11 12-12 what &lt;-&gt; explications When extracting phrase-to-phrase translations from the word-to-word alignment data we include a sufficient context which disambiguates the phrases. Given the word alignment data in example 1, the minimum context required to disambiguate the French word &amp;quot;explications&amp;quot; is the phrase 5-11 &lt;-&gt; 9-12. From the word alignment data we generate bilingual dictionaries in two different ways: a minimal dictionary contains only the shortest unambiguous phrase-torepresents the test data.</Paragraph>
      <Paragraph position="4"> As outlined in Melamed (1998), a sequence of words  which translates in a non-compositional fashion into a target sequence is exhaustively linked (see example 2). Phrase-to-phrase alignment Phrase-to-phrase alignment is represented by intervals indicating the starting and ending words of the phrases. In phrase-to-phrase alignment: (i) a sequence of English word tokens (i.e. a phrase) are mutually linked with sequences of French word tokens (i.e. a French phrase)</Paragraph>
      <Paragraph position="6"> We do not use the term &amp;quot;phrase&amp;quot; here in its linguistic sense: a phrase in this paper can be any sequence of words, even if they are not a linguistic constituent.</Paragraph>
      <Paragraph position="7"> phrase translations. For instance, from the alignment data in example 1, the following 8 entries are generated as a minimal dictionary:  As shorthand notation we use here the offset numbers. In the generated dictionary, we have extracted the sequences of words instead of the offset numbers.</Paragraph>
      <Paragraph position="8"> In an exhaustive dictionary all possible unambiguous phrase translations are extracted. An exhaustive dictionary is a superset of the minimal dictionary. For example 1, seven additional entries are generated:  Note that these additional phrase translations can be compositionally generated with the minimal dictionary. To evaluate the word alignment data through phrasal alignments, we generated three types of dictionaries for all 16 submissions and the test data: (i) an alignment-based minimal dictionary, align-dic1; actually 447 small dictionaries for each sample alignment.</Paragraph>
      <Paragraph position="9"> (ii) a text-based minimal dictionary (textdic1)which is the union of the align-dic1.</Paragraph>
      <Paragraph position="10"> (iii) an exhaustive text-based dictionary (textdic2) which is the union of exhaustive alignment dictionaries.</Paragraph>
      <Paragraph position="11"> As can be seen from figure 1, the size of the exhaustive dictionary (text-dic2) is in most cases much bigger than those of the minimal dictionaries align-dic1 and text-dic1. The reason is due to the way the data has been aligned.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>