<?xml version="1.0" standalone="yes"?>
<Paper uid="W05-0813">
  <Title>Symmetric Probabilistic Alignment</Title>
  <Section position="4" start_page="88" end_page="89" type="evalu">
    <SectionTitle>
3 Results and Conclusions
</SectionTitle>
    <Paragraph position="0"> Table 1 compares the performance of SPA on what is now the development data against the submissions with the best AER values reported by (Mihalcea and Pedersen, 2003) for the participants in the 2003 workshop, including CMU, MITRE, RALI, University of Alberta, and XRCE 1. As SPA generates only SURE alignments, the values in Table 1 are SURE alignments under the NO-NULL-Align scoring con- null word-to-word alignments needed for the shared task and was not tuned for this corpus, its performance is competitive with the best of the systems previously used for the shared task. We thus decided to submit runs for the official 2005 evaluation, whose resulting scores are shown in Table 2.</Paragraph>
    <Paragraph position="1"> On the development set, noncontiguous alignments resulted in slightly lower precision than contiguous alignments, which was not unexpected, but recall does not increase enough to improve F1 or AER. The modified dictionaries improved precision slightly, as anticipated, but lowered recall sufficiently to have no net effect on F1 or AER.</Paragraph>
    <Paragraph position="2"> The evaluation set proved to be very similar in difficulty to the development data, resulting in scores that were very close to those achieved on the dev-test set. Noncontiguous alignments again proved to have a very small negative effect on AER resulting from reduced precision, but this time the altered dictionaries for SPA(h) resulted in a substantial reduction in recall, considerably harming overall performance.</Paragraph>
    <Paragraph position="3"> After the shared task was complete, we performed some tuning of the alignment parameters for the Romanian-English development test set, and found that the French-English-tuned parameters were close to optimal in performance. The AER on the development test set for the SPA(c) contiguous alignments condition decreased from 36.44% to 36.11% after the re-tuning.</Paragraph>
  </Section>
  <Section position="5" start_page="89" end_page="89" type="evalu">
    <SectionTitle>
4 Future Work
</SectionTitle>
    <Paragraph position="0"> Enhancements in the extraction of word-to-word alignments from what is fundamentally a phrase-to-phrase alignment algorithm could probably further  improve results on the Romanian-English data. We also intend to investigate principled, seamless integration of manual alignments and dictionaries with probabilistic ones, since the ad hoc method proved detrimental. Finally, a more detailed performance analysis is in order, to determine whether the close balance of precision and recall is inherent in the bidirectionality of the algorithm or merely coincidence.</Paragraph>
  </Section>
class="xml-element"></Paper>