<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-0834">
  <Title>Supervised Word Sense Disambiguation with Support Vector Machines and Multiple Knowledge Sources</Title>
  <Section position="5" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
4 Evaluation
</SectionTitle>
    <Paragraph position="0"> Since our WSD system always outputs exactly one prediction for each test example, its recall is always the same as precision. We report below the micro-averaged recall over all test words.</Paragraph>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
4.1 Evaluation on SENSEVAL-2 and
SENSEVAL-1 Data
</SectionTitle>
      <Paragraph position="0"> Before participating in SENSEVAL-3, we evaluated our WSD system on the English lexical sample task of SENSEVAL-2 and SENSEVAL-1. The microaveraged, fine-grained recall over all SENSEVAL-2 test words and all SENSEVAL-1 test words are given in Table 2.</Paragraph>
      <Paragraph position="1"> In SENSEVAL-1, some example sentences are provided with the dictionary entries of the words used in the evaluation. We provide the recall on SENSEVAL-1 test data with and without the use of such additional dictionary examples in training.</Paragraph>
      <Paragraph position="2"> On both SENSEVAL-2 and SENSEVAL-1 test data, the accuracy figures we obtained, as reported in Table 2, are higher than the best official test scores reported on both evaluation data sets.</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
4.2 Official SENSEVAL-3 Scores
</SectionTitle>
      <Paragraph position="0"> We participated in the SENSEVAL-3 English lexical sample task, and both subtasks of the multilingual lexical sample task. The official SENSEVAL-3 scores are shown in Table 3. Each score is the micro-averaged recall (which is the same as precision) over all test words.</Paragraph>
      <Paragraph position="1"> According to the task organizers, the fine-grained (coarse-grained) recall of the best participating system in the English lexical sample task is 0.729 (0.795). As such, the performance of our system nusels compares favorably with the best participating system.</Paragraph>
      <Paragraph position="2"> We are not able to fully assess the performance of our multilingual lexical sample task systems nusmlst and nusmlsts at the time of writing this paper, since performance figures of the best participating system in this task have not been released by the task organizers.</Paragraph>
    </Section>
    <Section position="3" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
4.3 Utility of English Sense as an Additional
Knowledge Source
</SectionTitle>
      <Paragraph position="0"> To determine if using the English sense as an additional knowledge source improved the accuracy of the translation and sense subtask, we conducted a five-fold cross validation experiment. We randomly divided the training data of the translation and sense subtask for each word into 5 portions, using 4 portions for training and 1 portion for test. We then repeated the process by selecting a different portion as the test data each time and training on the remaining portions.</Paragraph>
      <Paragraph position="1"> Our investigation revealed that adding the English sense to the four existing knowledge sources improved the micro-averaged recall from 0.628 to 0.638 on the training data. As such, we decided to use the English sense as an additional knowledge source for our system nusmlsts.</Paragraph>
      <Paragraph position="2"> After the official SENSEVAL-3 evaluation ended, we evaluated a variant of our system nusmlsts without using the English sense as an additional knowledge source. Based on the official test keys released, the micro-averaged recall drops to 0.643, which seems to suggest that the English sense is a helpful knowledge source for the translation and sense subtask.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>