<?xml version="1.0" standalone="yes"?>
<Paper uid="W05-0801">
  <Title>Identifying Word Correspondences in Parallel Texts. In Proceedings of the Speech and Natural</Title>
  <Section position="3" start_page="0" end_page="1" type="intro">
    <SectionTitle>
2 Data and Methodology for these
Experiments
</SectionTitle>
    <Paragraph position="0"> The experiments reported here were carried out using data from the workshop on building and using parallel texts held at HLT-NAACL 2003 (Mihalcea and Pedersen, 2003). For the majority of our experiments, we used a subset of the Canadian Hansards bilingual corpus supplied for the workshop, comprising 500,000 English-French sentences pairs, including 37 sentence pairs designated as &amp;quot;trial&amp;quot; data, and 447 sentence pairs designated as test data. The trial and test data have been manually aligned at the word level, noting particular pairs of words either as &amp;quot;sure&amp;quot; or &amp;quot;possible&amp;quot; alignments. As an additional test, we evaluated our best alignment method using the workshop corpus of approximately 49,000 English-Romanian sentences pairs from diverse sources, including 248 manually aligned sentence pairs designated as test data.</Paragraph>
    <Paragraph position="1">  We needed annotated development data to optimize certain parameters of our algorithms, and we were concerned that the small number of sentence pairs designated as trial data would not be enough for this purpose. We therefore randomly split each of the English-French and English-Romanian test data sets into two virtually equal subsets, by randomly ordering the test data pairs, and assigning alternate pairs from the random order to the two subsets. We used one of these subsets as a development set for parameter optimization, and held out the other for a final test set.</Paragraph>
    <Paragraph position="2"> We report the performance of various alignment algorithms in terms of precision, recall, and alignment error rate (AER) as defined by Och and Ney (2003):</Paragraph>
    <Paragraph position="4"> In these definitions, S denotes the set of alignments annotated as sure, P denotes the set of alignments annotated possible or sure, and A denotes the set of alignments produced by the method under test. Following standard practice in the field, we take AER, which is derived from F-measure, as the primary evaluation metric that we are attempting to optimize.</Paragraph>
    <Paragraph position="5"> Our initial experiments involve algorithms that do not consider the positions of words in the sentences.</Paragraph>
    <Paragraph position="6"> Thus, they are incapable of distinguishing among multiple instances of the same word type in a sentence. We will say that these methods produce word type alignments. We compare these algorithms on the basis of the best possible alignment of word tokens given an alignment of word types. We go on to consider various ways of choosing a word token alignment for a given word type alignment, and all our final evaluations are conducted on the basis of the alignment of individual word tokens.</Paragraph>
    <Paragraph position="7"> and the hand alignments of the words in the trial and test data were created by Franz Och and Hermann Ney (Och and Ney, 2003). The manual word alignments for the English-Romanian test data were created by Rada Mihalcea and Ted Pedersen.</Paragraph>
  </Section>
class="xml-element"></Paper>