<?xml version="1.0" standalone="yes"?>
<Paper uid="P06-4004">
  <Title>Valido: a Visual Tool for Validating Sense Annotations</Title>
  <Section position="6" start_page="14" end_page="14" type="evalu">
    <SectionTitle>
4 Evaluation
</SectionTitle>
    <Paragraph position="0"> We briefly report here an experiment on the validation of manual sense annotations with the aid of Valido. For more detailed experiments the reader can refer to Navigli (2006).</Paragraph>
    <Paragraph position="1"> 1,000 sentences were uniformly selected from the set of documents in the semantically-tagged SemCor corpus (Miller et al., 1993). For each sentence = w1w2 :::wk annotated in SemCor with the senses sw1sw2 :::swk (swi 2 Senses(wi);i 2 f1;2;:::;kg), we randomly identified a word wi 2 , and chose at random a different sense swi for that word, that is swi 2 Senses(wi) nfswig.</Paragraph>
    <Paragraph position="2"> In other words, we simulated in vitro a situation in which an annotator provides an appropriate sense and the other selects a different sense.</Paragraph>
    <Paragraph position="3"> We applied Valido with policy ( ) to the annotated sentences and evaluated the performance of the approach in suggesting the appropriate choice for the words with disagreement. The results are reported in Table 2 for nouns, adjectives, and verbs (we neglected adverbs as very few interconnections can be found for them).</Paragraph>
    <Paragraph position="4"> The experiment shows that evidences of inconsistency due to inappropriate annotations are provided with good precision. The overall F1 measure is 59:18%. The chance baseline is 50%.</Paragraph>
    <Paragraph position="5"> The low recall obtained for verbs, but especially for adjectives, is due to a lack of connectivity in the lexical knowledge base, when dealing with connections across different parts of speech.</Paragraph>
  </Section>
class="xml-element"></Paper>