<?xml version="1.0" standalone="yes"?>
<Paper uid="I05-3005">
  <Title>Morphological features help POS tagging of unknown words across language varieties</Title>
  <Section position="10" start_page="37" end_page="38" type="evalu">
    <SectionTitle>
5.5.8 Evaluation
</SectionTitle>
    <Paragraph position="0"> Table 8 shows our results using the standard maximum entropy forward feature selection algorithm; at each iteration we add the feature family that most significantly improves the log likelihood of the training data given the model. We seed the feature space search with the features in Model Lt+LLt. From this model, adding suffix information gives a 9.58% (absolute) gain on unknown word tagging. Subsequently adding in prefix makes unknown word accuracy go up to 63.66%. Our first result is that Chinese affixes are indeed informative for unknown words. On the right side of Table 8, we can see that this performance gain is derived from better tagging of common nouns, verbs, proper nouns, numbers and others. Because earlier work in many languages including Chinese uses these simple prefix and suffix features (Brants 2000, Ng and Low 2004) we consider this performance (63.66% on unknown words) as our baseline.</Paragraph>
    <Paragraph position="1"> Adding in the feature set CTBM gives another 12.47% (absolute) improvement on unknown words.</Paragraph>
    <Paragraph position="2"> With this feature, punctuation shows the largest tagging improvement. The CTBM feature helps to identify punctuation since all other characters have been seen in different morpheme table made from the training set. That is, for a given word the lack of CTBM features cues that the word is a punctuation mark. Also, while this feature set generally helps all tagsets, it hurts a bit on nouns.</Paragraph>
    <Paragraph position="3"> Adding in the ASBC feature sets yields another 1.23% and 1.48% (absolute) gains on unknown words. These two feature sets generally improve performance on all tagsets. Including the verb affix feature helps with common nouns and proper nouns, but hurts the performance on verbs. Overall, it yields 0.21% gain on unknown words. Finally, adding the radical feature helps the most on nouns, while subsequently adding in the name entity morphemes help to reduce the error on proper nouns by 2.50%. Finally, adding in feature length renders a 0.25% gain on unknown words. Commutatively, applying the feature sets results in an overall accuracy of 91.97% and an unknown word accuracy of 79.86%.</Paragraph>
    <Section position="1" start_page="37" end_page="38" type="sub_section">
      <SectionTitle>
5.6 Experiments with the training sets of
</SectionTitle>
      <Paragraph position="0"> variable sizes and varieties In this section, we compare our best model with the baseline model using different corpora size and language varieties in the training set. All the evaluations are reported on the test set, which has roughly equal amounts of data from XH, HKSAR, and SM.</Paragraph>
      <Paragraph position="1"> The left column of Table 9 shows that when we train a model only on a single language variety and test on a mixed variety data, our unknown word accuracy is 79.50%, which is 18.48% (absolute) better than the baseline. The middle column shows when the training set is composed of different varieties and hence looks like the test set, performance of both the base-line and our best model improves.</Paragraph>
      <Paragraph position="2">  The right column shows the effect of doubling the training set size, using mixed varieties. As expected, using more data benefits both models.</Paragraph>
      <Paragraph position="3"> These results show that having training data from different varieties is better than having data from one source. But crucially, our morphological-based features improve the tagging performance on unknown words even when the training set includes some data that resembles the test set.</Paragraph>
      <Paragraph position="4"> How good are our best numbers, i.e. 93.7% on POS tagging in CTB 5.0? Unfortunately, there are no clean direct comparisons in the literature. The closest result in the literature is Xue et al. (2002), who re-train the Ratnaparkhi (1996) tagger and reach accuracies of 93% using CTB-I. However CTB-I contains only XH data and furthermore the data split is no longer known for this experiment (Xue p.c.) so a comparison is not informative. However, our performance on tagging when trained on Training I and tested on just the XH part of the test set is 94.44%, which might be a more relevant comparison to Xue et al. (2002).</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>