<?xml version="1.0" standalone="yes"?>
<Paper uid="W96-0306">
  <Title>Acquisition of Semantic Lexicons: Using Word Sense Disambiguation to Improve Precision</Title>
  <Section position="3" start_page="42" end_page="43" type="metho">
    <SectionTitle>
2 Verb Classification Based on Syntactic Behavior
</SectionTitle>
    <Paragraph position="0"> We build on the syntactic filter approach of (Dorr, Garman, and Weinberg, 1995), in which verbs were automatically classified into semantic classes using syntactic encodings in LDOCE. This earlier approach produced a ranked assignment of verbs to the semantic classes from (Levin, 1993) based on syntactic tests (e.g., whether a verb occurs in a dative construction such as Mary gave John the book). 2 The syntactic approach alone was demonstrated to classify Levin verbs with 47% accuracy (i.e., 1812 correct verb classifications out of 3851 possible assignments).</Paragraph>
    <Paragraph position="1"> The measure of success used in the purely syntactic approach is flawed in that the &amp;quot;accuracy&amp;quot; factor was based on the number of correct assignments in the five top-ranked assignments produced by their algorithm. A better measure of the efficacy of the algorithm would be to examine the ratio of correct assignments to the total number of assignments. The algorithm in (Dorr, Garman, and Weinberg, 1995) is correct only 13% of the time (1812 correct assignments out of 13761 total assignments) if given up to 5 assignments per verb. If given up to 15 assignments, the situation 2Levin's semantic classes are labeled with numbers ranging from 9 to 57; the actual number of semantic classes is 191 (not 46) due to many class subdivisions under each major class, These 191 classes cover 2813 verbs that occur in the LDOCE. Since verbs may occur in multiple classes, the number of possible assignments of LDOCE verbs into classes is 3851.</Paragraph>
    <Paragraph position="2">  would deteriorate further: even though 2607 out of 3851 possible assignments would be correct, these correct assignments constitute only 6.5% of the total number of assignments made by the algorithm.</Paragraph>
    <Paragraph position="3"> We borrow terminology from Information Filtering (see, e.g., (Lewis, 1992)) to characterize these results. In particular, Recall is the number of correct categorizations the algorithm gives divided by the number of correct categorizations already given in the database. Precision, on the other hand, is the number of correct categorizations that the algorithm gives divided by the total number of categorizations that it gave. In these terms, the algorithm in (Dorr, Garman, and Weinberg, 1995) achieves a recall of 67.7%, but a precision of 6.5% if given up to 15 semantic class assignments per verb.</Paragraph>
    <Paragraph position="4"> In addition to low precision, the purely syntactic filter described above was tested only on verbs that are in (Levin, 1993) and it did not take into account the problem of multiple word senses. The remainder of this paper describes the formulation and refinement of semantic filters that increases the precision of this earlier experiment, while extending the coverage to novel verbs (i.e., ones not occurring in (Levin, 1993)) and addressing the polysemy problem.</Paragraph>
  </Section>
  <Section position="4" start_page="43" end_page="44" type="metho">
    <SectionTitle>
3 Semantic Filter: Increasing Precision
</SectionTitle>
    <Paragraph position="0"> We take as our starting point 7767 LDOCE verbs, approximately 5000 of which do not occur in Levin's classes. Each of these verbs was assigned up to 15 possible semantic classes, ranked by the degree of likelihood that the verb belongs to that class, giving a total of 113,106 ranked assignments.</Paragraph>
    <Paragraph position="1"> As described above, the syntactic filter discovers 2607 of the 3851 assignments of LDOCE verbs found in Levin's semantic classes. These assignments are particularly interesting because we know they are correct, and we can see how high the program ranks the correct assignments.</Paragraph>
    <Paragraph position="2"> To create a semantic filter, we take a semantic class from Levin and extend it with related verbs from WordNet. We call this extended list a semantic field. Verbs that do not occur in the semantic field of a particular class fail to pass through the semantic filter for that class, by definition. We first examined different semantic relations provided by WordNet (synonymy, hyponymy, both synonyms and hyponyms, and synonyms of synonyms) in order to determine which one would be most appropriate for constructing semantic fields for each of Levin's 191 verb classes. We evaluated the performance of these different relations by examining the degree of class coverage of the relation using a prototypical verb from each class. 3 For example, the Change of State verbs of the break subclass (Class 45.1) contains the verbs break, chip, crack, crash, crush, fracture, rip, shatter, smash, snap, splinter, split, tear. The full semantic field contains the union of the related verbs for every verb in the original Levin class.</Paragraph>
    <Paragraph position="3"> Thus, if we build our semantic field on the basis of the synonymy relation, all synonyms of verbs in a particular class would be legal candidates for membership in that class. For Class 45.1, using the synonymy relation would result in a field size of 185 (i.e., there are 185 WordNet synonyms for the 13 verbs in the class); by contrast, the hyponymy relation would yield a field size of 245.</Paragraph>
    <Paragraph position="4"> To choose a relation to use for the semantic field, we looked at verbs semantically related to the prototypical verb in each class, and checked how many of the verbs in each class would be included in the filter. We examined several relations based on combinations of synonymy and hyponymy.</Paragraph>
    <Paragraph position="5"> We considered the best candidate to be the one that matched the greatest proportion of the verbs  for that class. These tests are based on grammaticahty of usage in certain well-defined contexts (e.g., the dative construction).</Paragraph>
    <Paragraph position="6">  prototype verb, matched an average of 20% of the Levin verbs, while having an average size of 11 verbs. The average size of Levin's semantic classes is 22 verbs.</Paragraph>
    <Paragraph position="7"> Let us now:look at the behavior of the synonymy-based semantic filter. Of the 113,106 assignments of LDOCE verbs to Levin classes given by the syntactic filter, 6029 (19%) pass through the semantic filter. Clearly, the semantic filter constrains the possible assignments, but the question to ask is whether the constraint improves the accuracy of the assignments. To answer this, we first examined the 2813 verbs in LDOCE that also appear in Levin to see if they matched Levin's categorization.</Paragraph>
    <Paragraph position="8"> Without the semantic filter, the syntactic filter provides up to 15 semantic-class assignments for each of the 2813 verbs, giving 40,248 assignments, as shown in Table 1. 2,607 of these assignments  (6.5%) are correct. When we add the semantic filter, the number of assignments drops to 4168, 10% of the unfiltered assignments. 2607 of these (62.5%) are correct, a twelve-fold improvement over the unfiltered assignments.</Paragraph>
    <Paragraph position="9"> By Right Assignments, we mean: cases in which the system assigns a verb to a given Levin  class, when that verb appears in that class in Levin's book. By Wrong Assignments, we mean: cases in which the system assigns a verb to a given Levin class, when that verb does not appear in that class in Levin's book.</Paragraph>
    <Paragraph position="10"> It is important to point out that even though the semantic filter is based on words in Levin, it still sometimes categorized the Levin verb incorrectly. Since the filter is based on synonyms of Levin verbs, in some cases, a synonym of a verb from some other class will appear in the set that does not belong there. In this case, there are 1561 assignments known to be wrong, out of a total of 4168 assignments. For example, the verb scatter is a synonym of break in WordNet. Because the verb break occurs in each of these classes, the semantic filter based on synonyms assigns scatter to classes 10.6 (Cheat Verbs), 23.2 (Split Verbs), 40.8.3 (Hurt Verbs), 45.1 (Break Verbs), 48.1.1 (Appear Verbs). But the correct class for scatter is 9.7 (Spray/Load Verbs). This illustrates the difficulty of using an approach that does not account for multiple word senses. We will address this point further in section 3.</Paragraph>
    <Paragraph position="11"> Setting aside the polysemy problem, we see that this semantic filter is very useful for reducing the number of incorrect assignments.</Paragraph>
  </Section>
  <Section position="5" start_page="44" end_page="45" type="metho">
    <SectionTitle>
4 Performance on Novel Words
</SectionTitle>
    <Paragraph position="0"> We now examine how well it performs on unknown words by constructing a semantic filter based on three different proportions of the original 2813 Levin verbs: (a) 50%, (b) 70%, and (c) 90%, chosen randomly. 4 We then checked whether the &amp;quot;unknown&amp;quot; verbs (those not used to construct  proportions of Levin verbs. Consider the rows that show the behavior of the experiment which uses 50% of Levin's verbs, and tries to guess the remaining verbs using synonymy. Recall that there are 2607 verbs all together. In this case, 1282 verbs were chosen at random to use in constructing the filter. We call these the &amp;quot;known&amp;quot; verbs. This leaves 1325 for use in evaluating the semantic filter--we call these the &amp;quot;novel&amp;quot; verbs. For the 1282 known verbs, the filter made 1752 assignments to semantic classes. There were 470 wrong assignments and 1282 right ones, giving a precision rate of 73.2% and recall rate of 100.0% .</Paragraph>
  </Section>
  <Section position="6" start_page="45" end_page="47" type="metho">
    <SectionTitle>
5 The Effect of Disambiguation
</SectionTitle>
    <Paragraph position="0"> As mentioned previously, the problem with the semantic filter we have defined is that it is not sensitive to multiple word senses of the particular verbs in the semantic classes. For example, there are 23 senses of the verb break in WordNet. This includes senses which correspond to the Change of State verbs, such as Sense 9, &amp;quot;break, bust, cause to break&amp;quot;, the synonyms of which are destroy, ruin, bust up, wreck, wrack. But it also includes irrelevant senses, such as Sense 7, &amp;quot;break dance&amp;quot;, the synonyms of which are dance, do a dance, perform a dance. Clearly, the semantic filter would behave better if we used word senses in creating the fields. As an attempt to address the polysemy problem, we conducted an exploratory study in which the verbs in Levin's semantic classes were disambiguated by hand: each verb received as many WordNet senses as were applicable.</Paragraph>
    <Paragraph position="1"> The performance of the various filters is shown in Table 3. To see the effect of disambiguation, compare the difference between undisambiguated and disambiguated synonyms. Precision has increased from 62.5% to 85.3%. For novel verbs, in the experiment which uses 50% of the verbs and  tries to guess the rest, the precision increases from 49.0% to 70.8%. But notice also that the recall decreases: with disambiguation (in the 50% study), recall drops from 31.1% for undisambiguated verbs to 21.6% for disambiguated verbs. The reason for this is that the undisambiguated filters contain numerous assignments which are correct but are included only accidentally.</Paragraph>
    <Paragraph position="2"> Table 3 also shows the performance of two other semantic filters based on hyponyms. We found that using hyponyms of hypernyms (going up one level in abstraction, and then one level back down) gave much better recall than plain synonymy, although the precision is lower. We also built a filter based on the union of synonyms with hyponyms of hypernyms. The effect of the synonyms on this filter was negligible, presumably since synonyms are often hyponyms of hypernyms. The results for both of these filters are shown in Table 3.</Paragraph>
  </Section>
class="xml-element"></Paper>