<?xml version="1.0" standalone="yes"?>
<Paper uid="W06-3504">
  <Title>Increasing the coverage of a domain independent dialogue lexicon with VERBNET</Title>
  <Section position="9" start_page="29" end_page="30" type="evalu">
    <SectionTitle>
5 Evaluation and discussion
</SectionTitle>
    <Paragraph position="0"> Since our goal in this evaluation is to balance the coverage of VERBNET with precision, we correspondingly evaluate along those two dimensions.</Paragraph>
    <Paragraph position="1"> For both techniques, we evaluate how many word senses were added, and the number of different words defined and VERBNET classes covered. As a measure of precision we use, for those entries which were retrieved, the percentage of those which could be taken &amp;quot;as is&amp;quot; (good entries) and the percentage of entries which could be taken with minor edits (for example, changing an LF type to a more specific subclass, or changing a semantic role in a template).</Paragraph>
    <Paragraph position="2"> The results of evaluation are shown in Table 2.7 Since for mapping with syntax filtering we considered all possible TRIPS-VERBNET intersections, it in effect presents an upper bound the number of words shared between the two databases. Further 7&amp;quot;nocos&amp;quot; table rows exclude the other cos VERBNET class, which is exceptionally broad and skews evaluation results.  extension would require extending the TRIPS LF Ontology with additional types to cover the missing classes. As can be seen from this table, 65% of VERBNET classes have an analogous class in TRIPS. At the same time, there is a very large number of class intersections possible, so if all possible intersections are generated, only a very small percentage of generated word senses (16%) is usable in the combined system. Thus developing techniques to filter out the irrelevant senses and class matches is important for successful hand-checking.</Paragraph>
    <Paragraph position="3"> Our evaluation also shows that while class intersection with thresholding provides higher precision, it does not capture many words and verb senses. One reason for this is data sparsity. TRIPS is relatively small, and both TRIPS and VERBNET contain a number of 1-word classes, which cannot be reliably mapped without human intervention. This problem can be alleviated in part as the size of the database grows. We expect this technique to have better recall when the combined lexicon is used to merge with a different lexical database such as FRAMENET.</Paragraph>
    <Paragraph position="4"> However, a more difficult issue to resolve is differences in class structure. VERBNET was built around the theory of syntactic alternations, while TRIPS used FRAMENET structure as a starting point, simplifying the role structure to make connection to parsing more straightforward (Dzikovska et al., 2004). Therefore TRIPS does not require that all words associated with the same LF type share syntactic behaviour, so there are a number of VERBNET classes with members which have to be split between different TRIPS classes based on additional semantic properties. 70% of all good matches in the filtering technique were such partial matches. This significantly disadvantages the thresholding technique, which provides the mappings on class level, not allowing for splitting word entries between the classes.</Paragraph>
    <Paragraph position="5"> We believe that the best solution can be found by combining these two techniques. The thresholding technique could be used to establish reliable class mappings, providing classes where many entries could be transferred &amp;quot;as is&amp;quot;. The mapping can then be examined to determine incorrect class mappings as well as the cases where classes should be split based on individual words. For those entries judged reliable in the first pass, the syntactic structure can be transferred fully and quickly, while the syntactic filtering technique, which requires more manual checking, can be used to transfer other entries in the intersections where class mapping could not be established reliably.</Paragraph>
    <Paragraph position="6"> Establishing class and member correspondence is a general problem with merging any two semantic lexicons. Similar issues have been noted in comparing FRAMENET and VERBNET (Baker and Ruppenhofer, 2002). A method recently proposed by Kwon and Hovy (2006) aligns words in different semantic lexicons to WordNet senses, and then aligns semantic roles based on those matches. Since we are designing a lexicon for semantic interpretation, it is important for us that all words should be associated with frames in a shared hierarchy, to be used in further interpretation tasks. We are considering using this alignment technique to further align semantic classes, in order to produce a shared database for interpretation covering words from multiple sources.</Paragraph>
  </Section>
class="xml-element"></Paper>