<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-0861">
  <Title>The &amp;quot;Meaning&amp;quot; System on the English Allwords Task</Title>
  <Section position="6" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
5 Evaluation on the Senseval-3 Corpus
</SectionTitle>
    <Paragraph position="0"> The Senseval-3 test set contains 2,081 target words, 1,851 of them polysemous. The subset covered by the SemCor-1.6 training contains 1,211 target words (65.42%, compared to the 56.0% of the Senseval-2 corpus). We submitted the outputs of two different configurations of the Meaning system: Meaningc and Meaning-wv. These systems correspond to Base-3 and W-Vot (in the best configuration) from table 3, respectively. The results from the official evaluation are given in table 4. Again, we applied an automatic mapping from WordNet-1.6 to WordNet1.7.1 synset labels. However, there are senses in 1.7.1 that do not exist in 1.6, thus our system simply cannot assign them.</Paragraph>
    <Paragraph position="1"> It can be observed that, even though on the tuning corpus both variants obtained very similar precision (67.4 and 67.5), on the test set the weighted voting scheme is clearly better than the baseline system, probably due to the robustness achieved by the ensemble. The performance decrease observed on the test set with respect to the Senseval-2 corpus is very significant (a0 5 points). Given that the baseline system performs worse than the voted approach, it seems unlikely that there is overfitting during the ensemble tuning. However, we plan to repeat the tuning experiments directly on the Senseval-3 corpus to see if the same behavior and conclusions are observed. Probably, the decrease in performance is due to the differences between the training and test corpora. We intend to investigate the differences between SemCor-1.6, Senseval-2, and Senseval-3 corpora at different levels of linguistic information in order to check the appropriateness of using SemCor-1.6 as the main information source.</Paragraph>
  </Section>
class="xml-element"></Paper>