<?xml version="1.0" standalone="yes"?>
<Paper uid="W97-0212">
  <Title>Sense Tagging in Action Combining Different Tests with Additive Weightings</Title>
  <Section position="6" start_page="76" end_page="76" type="evalu">
    <SectionTitle>
5. Evaluation
</SectionTitle>
    <Paragraph position="0"> These results clearly need to be improved dramatically before automatic sense tagging can prove practically useful. Nonetheless, these results, especially at sub-sense level, compare favourably with other research in the area.</Paragraph>
    <Paragraph position="1"> Ng and Lee (1996) have found only 57% agreement when comparing the same texts tagged according to the same dictionary senses by different (human!) research groups. Cowie, Guthrie and Guthrie (1992) have reported 72% correct assignment at the LDOCE homograph level (and a much lower level for individual sense assignment). Wilks, Slator and Guthrie (1996) comment that 62% accuracy can be achieved at this level just by assigning the first (therefore most frequent) homograph in LDOCE. Furthermore, Wilks and Stevenson (1996) propose a method which should apparently achieve 92% accuracy to that same level just by using grammatical tags.</Paragraph>
    <Paragraph position="2"> It must be noted however that the LDOCE homograph level is far more rough-grained than the CIDE guideword level, let alone the sub-sense level, and that Wilks and Stevenson's approach on its own would, by its very nature, not transfer down to more fine-grained distinctions. Other research, such as Yarowsky's into accent restoration in Spanish and French (1994), which reports accuracy levels of 90%99%, is again at a more rough-grained level, in this case that of distinguished unaccented and accented word forms.</Paragraph>
    <Paragraph position="3"> While the sense tagging results are fairly encouraging, the part of speech tagging results arc at present relatively poor. It thus secrns sensible, especially noting Wilks and Stevenson's analysis mentioned above, to first run a sentence through a traditional part of speech tagger before trying to disambiguate the senses. In thcory, we would expect information such as subject domain and collocations to help part of speech tagging to be more accurate, however slightly, but we have not yet bccn able to demonstrate this in practice.</Paragraph>
  </Section>
class="xml-element"></Paper>