<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-0861">
  <Title>The &amp;quot;Meaning&amp;quot; System on the English Allwords Task</Title>
  <Section position="4" start_page="0" end_page="0" type="intro">
    <SectionTitle>
3 Evaluation of Individual Modules
</SectionTitle>
    <Paragraph position="0"> For simplicity, and also due to time constraints, the supervised modules were trained exclusively on the SemCor-1.6 corpus, intentionally avoiding the use of other sources of potential training examples, e.g, other corpora, WordNet examples and glosses, similar/substitutable examples extracted from the same Semcor-1.6, etc. An independent training set was generated for each polysemous word (of a certain part-of-speech) with 10 or more examples in the SemCor-1.6 corpus. This makes a total of 2,440 independent learning problems, on which all supervised WSD systems were trained.</Paragraph>
    <Paragraph position="1"> The feature representation of the training examples was shared between all learning modules. It consists of a rich feature representation obtained using the Feature Extraction module of the TALP team in the Senseval-3 English lexical sample task.</Paragraph>
    <Paragraph position="2"> The feature set includes the classic window-based pattern features extracted from a local context and the &amp;quot;bag-of-words&amp;quot; type of features taken from a broader context. It also contains a set of features representing the syntactic relations involving the target word, and semantic features of the surrounding words extracted from the MCR of the Meaning project. See (Escudero et al., 2004) for more details on the set of features used.</Paragraph>
    <Paragraph position="3"> The validation corpus for these classifiers was the Senseval-2 allwords dataset, which contains 2,473 target word occurrences. From those, 2,239 occurrences correspond to polysemous words. We will refer to this subcorpus as S2-pol. Only 1,254 words from S2-pol were actually covered by the classifiers trained on the SemCor-1.6 corpus. We will refer to this subset of words as the S2-pol-sup corpus. The conversion between WordNet-1.6 synsets (SemCor1.6) and WordNet-1.7 (Senseval-2) was performed on the output of the classifiers by applying an automatically derived mapping provided by TALP2.</Paragraph>
    <Paragraph position="4"> Table 1 shows the results (precision and coverage) obtained by the individual supervised modules on the S2-pol-sup subcorpus, and by the unsupervised modules on the S2-pol subcorpus (i.e., we exclude from evaluation the monosemous words).</Paragraph>
    <Paragraph position="5"> Support Vector Machines and AdaBoost are the best performing methods, though all of them perform in a small accuracy range from 53.4% to 59.5%.</Paragraph>
    <Paragraph position="6"> Regarding the unsupervised methods, DDD is clearly the best performing method, achieving a remarkable precision of 61.9% with the DDDa0 variant, at a cost of a lower coverage. The DDDa1a3a2 appears to be the best system for augmenting the coverage of the former. Note that the autoPS heuristic for ranking senses is a more precise estimator than the WordNet most-frequent-sense (MFS).</Paragraph>
  </Section>
class="xml-element"></Paper>