<?xml version="1.0" standalone="yes"?>
<Paper uid="C04-1112">
  <Title>A Lemma-Based Approach to a Maximum Entropy Word Sense Disambiguation System for Dutch</Title>
  <Section position="7" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
6 Results and Evaluation
</SectionTitle>
    <Paragraph position="0"> In order to be able to evaluate the results from the lemma-based approach, we also include results based on wordform classifiers. During training with wordform classifiers, 953 separate classifiers were built.</Paragraph>
    <Paragraph position="1"> With the lemma-based approach, 669 classifiers were built in total during training, 372 based on the lemma of an ambiguous word (subsuming 656 wordforms) and 297 based on the wordform. A total of 512 unique ambiguous wordforms was found in the test data. 438 of these were classified using the classifiers built from the training data, whereas only 410 could be classified using the wordform model (see table 1 for an overview).</Paragraph>
    <Paragraph position="2"> We include the accuracy of the WSD system on all words for which classifiers were built (ambig) as well as the overall performance on all words (all), including the non-ambiguous ones. This makes our results comparable to other systems which use the same data, but maybe a different data split or a different number of classifiers (e.g. in connection with a frequency threshold applied). The baseline has been computed by always choosing the most frequent sense of a given wordform in the test data. The results in table 2 show the average accuracy for the two different approaches. The accuracy of both approaches improves significantly (when applying a paired sign test with a confidence level of 95%) over the baseline. This demonstrates that the general idea of the system, to combine linguistic features with statistical classification, works well. Focusing on a comparison of the two approaches, we can clearly see that the lemma-based approach works significantly better than the wordform only model, thereby verifying our hypothesis.</Paragraph>
    <Paragraph position="3"> Another advantage of the approach proposed, besides increasing the classification accuracy, is that less classifiers need to be built during training and therefore the WSD system based on lemmas is smaller. In an online application, this might be an important aspect of the speed and the size of the application. It should be noted here that the degree of generalization through lemmatization strongly depends on the data. Only inflected wordforms occurring in the corpus are subsumed in one lemma classifier. The more different inflected forms the training corpus contains, the better the &amp;quot;compression rate&amp;quot; in the WSD model. Added robustness is a further asset of our system. More wordforms could be classified with the lemma-based approach compared to the wordform-based one (438 vs. 410).</Paragraph>
    <Paragraph position="4"> In order to better assess the real gain in accuracy from the lemma-based model, we also evaluated a subpart of the results for the lemma-based and the wordform-based model, namely the accuracy of those wordforms which were classified based on their lemma in the former approach, but based on their wordform in the latter case. The comparison in table 3 clearly shows that there is much to be gained from lemmatization. The fact that inflected wordforms are subsumed in lemma classifiers leads to an error rate reduction of 8% and a system with less than half as many classifiers.</Paragraph>
    <Paragraph position="5"> In table 4, we see a comparison with another WSD systems for Dutch which uses Memory-Based learning (MBL) in combination with local context (Hendrickx et al., 2002). A big difference with the system presented in this article is that extensive parameter optimization for the classifier of each ambiguous wordform has been conducted for the MBL approach. Also, a frequency threshold of minimally 10 training instances was applied, using the baseline classifier for all words below that threshold. As we can see, our lemma-based WSD system scores the same as the Memory-Based WSD system, without extensive &amp;quot;per classifier&amp;quot; parameter optimization. According to Daelemans and Hoste (2002), different machine learning results should be compared once all parameters have been optimized for all classifiers. This is not the case in our system, and yet it achieves the same accuracy as an optimized model. Optimization of parameters for each ambiguous wordform and lemma classifier might help increase our results even further.</Paragraph>
  </Section>
class="xml-element"></Paper>