<?xml version="1.0" standalone="yes"?>
<Paper uid="H05-2007">
  <Title>Pattern Visualization for Machine Translation Output</Title>
  <Section position="3" start_page="0" end_page="12" type="metho">
    <SectionTitle>
2 Part-of-Speech Sequence Recall
</SectionTitle>
    <Paragraph position="0"> In developing a new analysis method, we are motivated in part by recent studies suggesting that word reorderings follow general patterns with respect to syntax, although there remains a high degree of flexibility (Fox, 2002; Hwa et al., 2002). This suggests that in a comparative analysis of two MT systems (or two versions of the same system), it may be useful to look for syntactic patterns that one system (or version) captures well in the target language and the other does not, using a syntaxbased, recall-oriented metric.</Paragraph>
    <Paragraph position="1"> As an initial step, we would like to summarize reordering patterns using part-of-speech sequences. Unfortunately, recent work has confirmed the intuition that applying statistical analyzers trained on well-formed text to the noisy output of MT systems produces unuseable results (e.g. (Och et al., 2004)). Therefore, we make the conservative choice to apply annotation only to the reference corpus. Word n-gram correspondences with a reference translation are used to infer the part-of-speech tags for words in the system output.</Paragraph>
    <Paragraph position="2"> The method:  1. Part-of-speech tag the reference corpus. We used  interface uses color to make examples easy to find.</Paragraph>
    <Paragraph position="3"> MXPOST (Ratnaparkhi, 1996), and in order to discover more general patterns, we map the tag set down after tagging, e.g. NN, NNP, NNPS and NNS  all map to NN.</Paragraph>
    <Paragraph position="4"> 2. Compute the frequency freq(ti ...tj) of every possible tag sequence ti ...tj in the reference corpus. 3. Compute the correspondence between each hypoth- null esis sentence and each of its corresponding reference sentences using an approximation to maximum matching (Melamed et al., 2003). This algorithm provides a list of runs or contiguous sequences of words ei ...e j in the reference that are also present in the hypothesis. (Note that runs are order-sensitive.) 4. For each recalled n-gram ei ...e j, look up the associated tag sequence ti ...tj and increment a counter recalled(ti ...tj) Using this method, we compute the recall of tag patterns, R(ti ...tj) = recalled(ti ...tj)/freq(ti ...tj), for all patterns in the corpus.</Paragraph>
    <Paragraph position="5"> To compare two systems (which could include two versions of the same system), we identify POS n-grams that are recalled significantly more frequently by one system than the other, using a difference-of-proportions test to assess statistical significance. We have used this method to analyze the output of two different statistical machine translation models (Chiang et al., 2005).</Paragraph>
  </Section>
  <Section position="4" start_page="12" end_page="12" type="metho">
    <SectionTitle>
3 Visualization
</SectionTitle>
    <Paragraph position="0"> Our demonstration system uses an HTML interface to summarize the observed pattern recall. Based on frequent or significantly-different recall, the user can select and visually inspect color-coded examples of each pattern of interest in context with both source and reference sentences. An example visualization is shown in Figure 1.</Paragraph>
  </Section>
  <Section position="5" start_page="12" end_page="12" type="metho">
    <SectionTitle>
4 Acknowledgements
</SectionTitle>
    <Paragraph position="0"> The authors would like to thank David Chiang, Christof Monz, and Michael Subotin for helpful commentary on this work. This research was supported in part by ONR MURI Contract FCPO.810548265 and Department of Defense contract RD-02-5700.</Paragraph>
  </Section>
class="xml-element"></Paper>