<?xml version="1.0" standalone="yes"?>
<Paper uid="C96-1055">
  <Title>Role of Word Sense Disambiguation in Lexical Acquisition: Predicting Semantics from Syntactic Cues</Title>
  <Section position="4" start_page="322" end_page="324" type="metho">
    <SectionTitle>
3 Syntax-Semantics Relation: -Verb
</SectionTitle>
    <Paragraph position="0"> Classification Based on Syntactic Behavior The central thesis of (Levin, 1993) is that the semantics of a verb and its syntactic behavior are predictably related. As a demonstration that such predictable relationships are not confined to an insignificant portion of the vocabulary, Levin surveys 4183 verbs, grouped into 191 semantic classes in Part Two of her book. The syntactic behavior of these classes is illustrated with 1668 example sentences, an average of 8 sentences per (:lass. Given the scope of bevin's work, it is not easy to verify the central thesis. 'lb this end, we created a database of Levin's verb classes and example sentences from each class, and wrote a parser to extract, basic syntactic patterns from tire sentences.1 We then characterized each semantic class by a set of syntactic patterns, which we call a syntactic signature, and used the resuiting database as the basis of two experiments, both designed to to discover whether the syntactic signatures tell us anything about the meaning of the verbs. 2 '\['he first experiment, which we label Verb-Based, ignores word-sense distinctions by assigning one syntactic signature to each verb, regardless of whether it occurred in multiple classes. The second experiment, which we label Class-Based, implicitly takes word-sense distinctions into account by considering each occurrence of a verb individually and assigning it a single syntactic signature according to class membership.</Paragraph>
    <Paragraph position="1"> The remainder of this section describes the assignrnent of signatures to semantic cbusses and the two experiments for determining the relation of syntactic information to semantic cbtsses. We will see that our classitication technique shows a 15-fold improvement in the experiment where we implicitly account for word-sense distinctions.</Paragraph>
    <Paragraph position="2">  of (Dubois and Saint-Dizier, 1995). In particular, we depart from the alternation-based data in (Levin, 1993), which is primarily binary in that sentences are presented in pairs which constitute an alternation. Following Saint-Dizier's work, we construct N-ary syntactic characterizations. The choice is of no empirieM consequence, but it simplifms the experiment by eliminating the problem of naming the syntactic patterns.</Paragraph>
    <Paragraph position="3"> Verbs: break, chip, crack, crash, crush, fracture, rip, shatter, slnash, snap, sl)linter, split, tear</Paragraph>
    <Section position="1" start_page="322" end_page="322" type="sub_section">
      <SectionTitle>
Example Sentences:
</SectionTitle>
      <Paragraph position="0"> Crystal vases break easily.</Paragraph>
      <Paragraph position="1"> The hammer broke the window.</Paragraph>
      <Paragraph position="2"> The window broke.</Paragraph>
      <Paragraph position="3"> q'ony broke her arm.</Paragraph>
      <Paragraph position="4"> 'l?ony broke his finger.</Paragraph>
      <Paragraph position="5"> &amp;quot;lbny broke the crystal vase. qbny broke the cup against the wall. q'ony broke the glass to 1)ieces. Tony broke the piggy bank open. Tony broke the window with a hanuner. Tony broke the window.</Paragraph>
      <Paragraph position="6"> *Tony broke at tit(; window.</Paragraph>
      <Paragraph position="7"> *qbny broke herself on the arm.</Paragraph>
      <Paragraph position="8"> *Tony broke himself.</Paragraph>
      <Paragraph position="9"> *qbny broke the wall with the cup. A break.</Paragraph>
      <Paragraph position="11"/>
    </Section>
    <Section position="2" start_page="322" end_page="322" type="sub_section">
      <SectionTitle>
3.1 Asslgntnent of Signatures
</SectionTitle>
      <Paragraph position="0"> For tile first experiment below, we construct a verb-based syntactic signature, while for the second exl)eriment, we constructed a class-based signature.</Paragraph>
      <Paragraph position="1"> The first step for constructing a signature is to decide what syntactic information to extract for ttre t)asic syntactic patterns that make up the signature.</Paragraph>
      <Paragraph position="2"> It turns out that a very simple strategy works well, namely, flat parses that contain lists of the major categories in the sentence, the verb, and a handfifl of other elements. The &amp;quot;parse&amp;quot;, then, for the sentence Tony broke the crystal vase is simply the syntactic pattern \[np,v,np\]. For Tony broke the vase to pieces we get \[np,v,np,pp(to)\]. Note that the pp node is marked with its head preposition. Table l shows an example class, the break subclass of the Change of State verbs (45.1), along with example sentences and the derived syntactic signature based on sentence patterns. Positive example sentences are denoted by the number 1 in the sentence patterns and negative example sentences are denoted by the number 0 (corresponding to sentences marked with a *).</Paragraph>
    </Section>
    <Section position="3" start_page="322" end_page="323" type="sub_section">
      <SectionTitle>
3.2 Experiment 1: Verb-based Approach
</SectionTitle>
      <Paragraph position="0"> In the first experiment, we ignored word sense distinctions and considered each verb only once, regardless of whether it occurred in multiple classes. In fact;, 46% of the verbs appear more than once. In some cases, the verb appears to have a related sense even though it appears in different classes. For example, the verb roll appears in two subclasses of Manner of Motion Verbs that are distinguished on the basis of whether the grammatical subject is animate or inanimate. In other cases, tile verb may have (largely) unrelated senses. For example, the verb move is both a Manner of Motion verb  and verb of Psychological State.</Paragraph>
      <Paragraph position="1"> To compose the syntactic signatures for each verb, we collect all of the syntactic patterns associated with every class a particular verb appears in, regardless of the different classes are semantically related. A syntactic signature for a verb, by definition, is the union of the frames extracted from every example sentence for each verb. The outline of the verb-based experiment is as follows:  1. Automatically extract syntactic information from the example sentences.</Paragraph>
      <Paragraph position="2"> 2. Group the verbs according to their syntactic signature. 3. Determine where the two ways of grouping verbs overlap: null (a) the semantic classification given by Levin.</Paragraph>
      <Paragraph position="3"> (1)) the syntactic classification based on the derived  syntactic signatures.</Paragraph>
      <Paragraph position="4"> To return to the Change of State verbs, we now consider the syntactic signature of the verb break, rather than the signature of the semantic class as a unit. The verb break belongs not only to the Change of State class, but also four other classes: 10.6 Cheat, 23.2 Split, 40.8.3 Hurl, and 48.1.1 Appear. Each of these classes is characterized syntactically with a set of sentences. The union of the syntactic patterns corresponding to these sentences forms the syntactic signature for the verb. So although the signature for the Change of State class has 13 frames, the verb break has 39 frames from the other classes it appears in.</Paragraph>
      <Paragraph position="5"> Conceptually, it is helpful to consider the difference between the intension of a function versus its extension. In this case, we are interested in the functions that group the verbs syntactically and semantically. Intensionally speaking, the definition of the function that groups verbs semantically would have something to do with the actual meaning of the verbs. ~ Likewise, the intension of the function that groups verbs syntactically would be defined in terms of something strictly syntactic, such as subcategorization frames. But the intensions of these functions are matters of significant theoretical investigation, and although much has been accomplished in this ~rea, the question of mapping syntax to semantics and vice versa is an open research topic. Therefore, we can turn to the extensions of the functions: the actual groupings of verbs, based on these two separate criteria. The semantic extensions are sets of verb tokens, and likewise, the syntactic extensions are sets of verb tokens. To the extent that these functions map between syntax and semantics intensionally, they will pick out the same verbs extensionally.</Paragraph>
      <Paragraph position="6"> So for the verb-based experiment, our technique for establishing the relatedness between the syntactic signatures and the semantic classes, is mediated by the verbs themselves. We compare the two orthogonal groupings of the inventory of verbs: the semantic classes defined by Levin and the sets of verbs that correspond to each of the derived syntactic signatures. When these two groupings overlap, we have discovered a mapping from the syntax of the verbs to their semantics, via the verb tokens. More specifically, we define the overlap index 3An example of the intensional characterization of the Levin classes are the definitions of Lexical Conceptual Structures which correspond to each of Levin's semantic classes. See (Dorr and Voss, to appear).</Paragraph>
      <Paragraph position="7"> as the number of overlapping verbs divided by the average of the number of verbs in the semantic class and the number of verbs in the syntactic signature. Thus an overlap index of 1.00 is a complete overlap and an overlap of 0 is completely disjoint. In this experiment, the sets of verbs with a high overlap index are of interest. When we parsed the 1668 example sentences in Part Two of Levin's book (including the negative examples), these sentences reduce to 282 unique patterns. The 191 sets of sentences listed with each of the 191 semantic classes in turn reduces to 748 distinct syntactic signatures. Since there are far more syntactic signatures than the 191 semantic classes, it is clear that the mapping between signatures and semantic classes is not direct,.</Paragraph>
      <Paragraph position="8"> Only 12 mappings have complete overlaps. That means 6.3% of the 191 semantic classes have a complete overlap with a syntactic signature.</Paragraph>
      <Paragraph position="9"> The results of this experiment are shown in Table 2.</Paragraph>
      <Paragraph position="10"> Three values are shown for each of the six variations in the experiment: the mean overlap, the median overlap, and the percentage of perfect overlaps (overlaps of value 1.00). In every case, the median is higher than the mean. Put another way, there is always a cluster of good overlaps, but the general tendency is to have fairly poor overlaps.</Paragraph>
      <Paragraph position="11"> The six variations of the experiment are as follows.</Paragraph>
      <Paragraph position="12"> The first distinction is whether or not to count the negative evidence. We note that the use of negative examples, i.e., plausible uses of the verb in contexts which are disallowed, was a key component of this experiment. There are 1082 positive examples and 586 negative examples. Although this evidence is useful, it is not available in dictionaries, corpora, or other convenient resources that could be used to extend Levin's classification. Thus, to extend our approach to novel word senses (i.e., words not occurring in Levin), we would not be able to use negative evidence. For this reason, we felt it necessary to determine the importance of negative evidence for building uniquely identifying syntactic signatures. As one might expect, throwing out the negative evidence degrades the usefulness of the signatures across the board. The results which had the negative evidence are shown in the left-hand column of numbers in Table 2, and the results which had only positive evidence are shown in the right-hand side.</Paragraph>
      <Paragraph position="13"> The second, three-way, distinction involves prepositions, and breaks the two previous distinctions involving negative evidence into three sub-cases. Because we were interested in the role of prepositions in the signatures, we also ran the experiment with two different parse types: ones that ignored the actual prepositions in the pp's, and ones that ignored all information except for the values of the prepositions. Interestingly, we still got useful results with these impoverished parses, although fewer semantic classes had uniquely-identifying syntactic signatures under these conditions. These results are shown in the three major rows of Table 2.</Paragraph>
      <Paragraph position="14"> The best result, using both positive and negative evidence to identify semantic classes, gives 6.3% of the verbs having perfect overlaps relating semantic classes to syntactic signatures. See Table 2 for the full results.</Paragraph>
    </Section>
    <Section position="4" start_page="323" end_page="324" type="sub_section">
      <SectionTitle>
3.3 Experiment 2&amp;quot; Class-based Approach
</SectionTitle>
      <Paragraph position="0"> In this experiment, we attempt to discover whether each class-based syntactic signature uniquely identifies a sin- null gle semantic class. By h)cnsing on the classes, the verbs are implicitly disambiguated: the word sense is by definition the sense of the verb as a member of a given class. To compare these signatures with the previous verb-based signatures, it may be helpfnl to note that a verb-based signature is the union of all of the class~ based signatures of the semantic classes that the verb appears m.</Paragraph>
      <Paragraph position="1"> 'Fhe outline for this class-based exl)eriment is as follows: null  1. Automatically extract syntactic information from tile example sentences to yMd the syntactic signatnre for the class.</Paragraph>
      <Paragraph position="2"> 2. Determine which semantic classes have uniquely null identifying syntactic signatures.</Paragraph>
      <Paragraph position="3"> If we use the class-based syntactic signatures containing t)rcposition-marked pp's and both positive and negative evidence, the 1668 example sentences reduce to 282 syntactic patterns, just as before. But now there are 189 class-based syntactic signatures, as compared with 748 verb-based signatures from before. 187 of them mriquely identify a semantic (:lass, meaning that 97.9% of the classes have uniquely identifying syntactic signatures. Four of the semantic classes do not have enough syntactic information to distinguish them uniquely. 4 Although the effects of the various distinctions were present in the verb-based experiment, these effects are much clearer in the class-based experiments. The effects of negative and positive evidence, as well as the three ways of handling prepositions show up much clearer here, as is clear in Table 4.</Paragraph>
      <Paragraph position="4"> In the class-based experiment, we counted the percentage of semantic classes that had uniquely ide.ntifying signatures. In the verb-based experiment, we counted the number of perfect overlaps (i.e., index of 1.00) between the verbs as grouped in the semantic classes and grouped by syntactic signature. The over-all results of the suite of experiments, illustrating tile role of disambiguation, negative evidence, and prepositions, is shown in Table 4. There were three ways of treating prepositions: (i) mark the pp with the preposition, (ii) ignore the preposition, and (iii) keel) only the prepositions. For these different strategies, we see the percentage of perfect overlaps, as well as both tire</Paragraph>
    </Section>
  </Section>
  <Section position="5" start_page="324" end_page="325" type="metho">
    <SectionTitle>
4 Semantic Classification of Novel
Words
</SectionTitle>
    <Paragraph position="0"> As we saw above, word sense disambiguation is critical to tile success of any \[exical acquisition algorithm. The Levin-based verbs are already disambiguated by virtue of their membership in different classes. The difficulty, then, is to disambiguate and classify verbs that do not occur in Levin. Our current direction is to make use of the results of tire first two experiments, i.e., the relation t)etween syntactic patterns and semantic classes, but to use two additional techniques for disambiguation and classification of non-Levin verbs: (1) extraction of synonym sets provided in WordNet (Miller, 1985), an online lexical database containing thesaurus-like relations such as synonymy; and (2) selection of appropriate synonyms based on correlations between syntactic information in l,ongman's Dictionary of Contemporary English (LDOCF,) (Procter, 1978) and semantic classes in Levin. 'Phe basic idea is to first determine tire most likely candidates for semantic classification of a verb by examining the verb's synonym sets, many of which intersect directly with the verbs classified by Leviu. The &amp;quot;closest&amp;quot; synonyms are then selected fl'om these sets by comparing the LDOCE grammar codes of tire unknown word with those associated with each synonym candidate. The use of LDOCE as a syntactic filter on tire semantics derived from WordNet is tire key to resolving word-sense ambiguity during the acquisition process. The fldl acquisition algorithm is as follows:  Given a verb, check I,evin (:lass.</Paragraph>
    <Paragraph position="1">  1. If in Levitt, classify directly.</Paragraph>
    <Paragraph position="2"> 2. if not in Levin, find synonym set from WordNet.</Paragraph>
    <Paragraph position="3"> (a) if synonym in Levin, select, the class that has the closest match with canonical LDOCE codes.</Paragraph>
    <Paragraph position="4"> (b) If no synonyms in Levin or canonical LDOCE  codes are completely mismatched, hypothesize new class.</Paragraph>
    <Paragraph position="5"> Note that this algorithm assmnes that there is a &amp;quot;canonicM&amp;quot; set of LDOCE codes tbr each of Levin's semantic classes. Table 5 describes the significance of a subset of the syntactic codes in LDOCE. (The total nmnber of codes is 174.) We have developed a relation between LDOCE codes and Levin classes, in mnch the same way that we associated syntactic signatures with the semantic classes in the earlier experiments. These canonical codes are for syntactic filtering (checking for the closest match) in the classification algorithm.</Paragraph>
    <Paragraph position="6"> As an example of how the word-sense disambiguation process and classifcation, consider the non-Levin verb attempt. The LDOCE specification for this verb is: T1 T3 T4 WV5 N. Using the synonymy feature of WordNet, the algorithm automatically extracts tire candidate classes associated with the synonyms of this word: (1) Class 29.6 &amp;quot;Masquerade Verbs&amp;quot; (ace), (2) Class 29.8 &amp;quot;Captain Verbs&amp;quot; (pioneer), (3) Class 31.1 &amp;quot;Amuse Verbs&amp;quot; (try), (4) Class 35.6 &amp;quot;Ferret Verbs&amp;quot; (seek), and (5) Class 55.2 &amp;quot;Complete Verbs&amp;quot; (initiate). The synonyms for each of these classes have the following LDOCE encodiugs, respectively: (1) I I-FOIl I-ON</Paragraph>
    <Paragraph position="8"> N. The largest intersection with the syntactic codes for attempt occurs with the verb try (TI T3 T4 N). However, Levin's class 31.1 is not the correct class for attempt since this sense of try has a &amp;quot;negative amuse&amp;quot; meaning (e.g., John's behavior tried my patience. In fact, the (:odes T1 'l'3 '1'4 are not part of the canonical class-code mapping associated with class 31.1. Thus, attempt falls under case 2(b) of the algorithm, and a new class is hypothesized. This is a case where word-sense disambiguation has allowed us to classify a new word and to enhance Levin's verb classification by adding a new class to the word try as well. In our experiment;s, our algorithm found severM additional non-Levin verbs that fell into this newly hypothesized (;lass, including aspire, attempt, dare, decide, desire, elect, need, and swear.</Paragraph>
    <Paragraph position="9"> We have automatically classified 10,000 &amp;quot;unknown&amp;quot; verbs, i.e., those not occurring in the Levin classification, using this technique. These verbs are taken from i e , translations provided in bilin- English &amp;quot;glosses&amp;quot; (.. ) . gual dictionaries for Spanish and Arabic) As a preliminary measure of success, we picked out 84 L1)OCE control vocabulary verbs, (i.e., primitive words used for defning dictionary entries) and hand-checked our results. We found that 69 verbs were classifed correctly, SThe Spanish-English dictionary was built at the University of Maryland; The Arabic-English dictionary was produced by Alpnet, a company in Utah that develops translation aids. We are Mso in the process of developing bilingual dictionaries for Korean and French, and we will be porting our LCS acquisition technology to these languages in the near future.</Paragraph>
    <Paragraph position="10"> i.e., 82% accuracy.</Paragraph>
  </Section>
  <Section position="6" start_page="325" end_page="325" type="metho">
    <SectionTitle>
5 Summary
</SectionTitle>
    <Paragraph position="0"> We have conducted two experiments with the intent of addressing the issue of word-sense ambiguity in extraction from machine-readable resources for the construe tion of large-scale knowledge sources. In the first experiment, verbs that appeared in different classes collected the syntactic information fl'om each class it appeared in. Therefore, the syntactic signature was coml)osed from all of the example sentences fi'om every (:lass the verb appeared in. In some cases, the verbs were seanantically unrelated and consequently the mat)ping from syntax to semantics was muddied. '\['he second experiment attelnpted to determine a relationship between a semantic class and the syntactic information associated with each class. Not surprisingly, but not insignificantly, this relationship was very clear, since this experiment avoided the problem of word sense ambiguity.</Paragraph>
    <Paragraph position="1"> These experiments served to validate Levin's claim that verb semantics and syntactic behavior are predictably related and also demonstrated that a significant con&gt; ponent of any lexical acquisition program is the ability to perform word-sense disambiguation.</Paragraph>
    <Paragraph position="2"> We have used the results of our first two experiments to help in constructing and augmenting online dictionaries for novel verb senses. We have used the same syntactic signatures to categorize new verbs into Lcvin's classes on the basis of WordNet and 1,1)O(?1!3. We are currently porting these results to new languages using online bilingual lexicons.</Paragraph>
  </Section>
class="xml-element"></Paper>