<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-2807">
  <Title>Different Sense Granularities for Different Applications</Title>
  <Section position="3" start_page="0" end_page="0" type="metho">
    <SectionTitle>
2 Background
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
2.1 Propbank
</SectionTitle>
      <Paragraph position="0"> PropBank [Kingsbury &amp; Palmer, 2002] is an annotation of the Wall Street Journal portion of the Penn Treebank II [Marcus, 1994] with dependency structures (or `predicate-argument' structures), using sense tags for highly polysemous words and semantic role labels for each dependency. An important goal is to provide consistent semantic role labels across different syntactic realizations of the same verb, as in the window in [  The window] broke.</Paragraph>
      <Paragraph position="1"> PropBank can provide frequency counts for (statistical) analysis or generation components in a machine translation system, but provides only a shallow semantic analysis in that the annotation is close to the syntactic structure and each verb is its own predicate.</Paragraph>
      <Paragraph position="2"> In addition to the annotated corpus, PropBank provides a lexicon that lists, for each broad meaning of each annotated verb, its Frameset, i.e., the possible arguments in the predicate and their labels and all possible syntactic realizations. The notion of ``meaning'' used is fairly coarse-grained, and it is typically motivated from differing syntactic behavior. The Frameset also includes a ``descriptor'' field for each role which is intended for use during annotation and as documentation, but which does not have any theoretical standing. The collection of Frameset entries for a verb is referred to as the verb's frame. As an example of a PropBank entry, we give the frame for the verb leave below. Currently, there are frames for over 3,000 verbs, with a total of just over 4,300 Framesets described. Of these 3,000 verb frames, only a small percentage 21.8 % (700) have more than one Frameset, with less than 100 verbs with 4 or more.</Paragraph>
      <Paragraph position="3"> The process of sense-tagging the PropBank corpus with the Frameset tags has just been completed.</Paragraph>
      <Paragraph position="4"> The criteria used for the Framesets are primarily syntactic and clear cut. The guiding principle is that two verb meanings are distinguished as different framesets if they have distinct subcategorization frames. For example, the verb 'leave' has 2 framesets with the following frames, illustrated by the examples in (1) and (2):  (1) John left the room.</Paragraph>
      <Paragraph position="5"> (2) Mary left her daughter-in-law her pearls in her will</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
2.2 WordNet Sense Groupings
</SectionTitle>
      <Paragraph position="0"> In a separate project, as part of Senseval tagging exercises, we have developed a lexicon with another level of coarse-grained distinctions, as described below.</Paragraph>
      <Paragraph position="1"> The Senseval-1 workshop (Kilgarriff and Palmer, 2000) provided convincing evidence that supervised automatic systems can perform word sense disambiguation (WSD) satisfactorily, given clear, consistent sense distinctions and suitable training data. However, the Hector lexicon that was used as the sense inventory was very small and under proprietary constraints, and the question remained whether it was possible to have a publicly available, broad-coverage lexical resource for English and other languages, with the requisite clear, consistent sense distinctions. null Subsequently, the Senseval-2 (Edmonds and Cotton, 2001) exercise was run, which included WSD tasks for 10 languages. A concerted effort was made to use existing WordNets as sense inventories because of their widespread popularity and availability. Each language had a choice between the lexical sample task and the all-words task. The most polysemous words in the English Lexical Sample task are the 29 verbs, with an average polysemy of 16.28 senses using the pre-release version of WordNet 1.7. Double blind annotation by two linguistically trained annotators was performed on corpus instances, with a third linguist adjudicating between inter-annotator differences to create the &amp;quot;Gold Standard.&amp;quot; The average inter-annotator agreement rate was only 71%, which is comparable to the 73% agreement for all words in SemCor, with a much lower average polysemy.</Paragraph>
      <Paragraph position="2"> However, a comparison of system performance on words of similar polysemy in Senseval-1 and Senseval-2 showed very little difference in accuracy (Palmer et al., submitted). In spite of the lower inter-annotator agreement figures for Senseval-2, the double blind annotation and adjudication provided a reliable enough filter to ensure consistently tagged data with WordNet senses.</Paragraph>
      <Paragraph position="3"> Even so, the high polysemy of the WordNet 1.7 entries on average poses a challenge for automatic word sense disambiguation. In addition, WordNet only gives a flat listing of alternative senses, unlike most standard dictionaries which are more structured and often provide hierarchical entries. To address this lack, the verbs were grouped by two or more people, with differences being reconciled, and the sense groups were used for coarse-grained scoring of the systems.</Paragraph>
      <Paragraph position="4"> The criteria used for groupings included syntactic and semantic ones. Syntactic structure performed two distinct functions in our groupings. Recognizable alternations with similar corresponding predicate-argument structures were often a factor in choosing to group senses together, as in the Levin classes and PropBank, whereas distinct subcategorization frames were also often a factor in putting senses in separate groups. Furthermore, senses were grouped together if they were more specialized versions of a general sense. The semantic criteria for grouping senses separately included differences in semantic classes of arguments (abstract versus concrete, animal versus human, animacy versus inanimacy, different instrument types...), differences in the number and type of arguments (often reflected in the subcategorization frame as discussed above), differences in entailments (whether an argument refers to a created entity or a resultant state), differences in the type of event (abstract, concrete, mental, emotional...), whether there is a specialized subject domain, etc.</Paragraph>
      <Paragraph position="5"> Senseval-2 verb inter-annotator disagreements were reduced by more than a third when evaluated against the groups, from 29% to 18%, and by over half in a separate study, from 28% to 12%. A similar number of random groups provided almost no benefit to the inter-annotator agreement figures (74% instead of 71%), confirming the greater coherence of the manual groupings.</Paragraph>
      <Paragraph position="6"> 3 Mapping of Sense Groups to Framesets Groupings of senses for Senseval-2, as discussed above, use both syntactic and semantic criteria. Propbank, on the other hand, uses mostly syntactic cues to divide verb senses into framesets. As a result, framesets are more general than sense-groups and usually incorporate several sense groups. We have been investigating whether or not the groups developed for SENSEVAL-2 can provide an intermediate level of hierarchy in between the Prop-Bank Framesets and the WN 1.7 senses, and our initial results are promising. Based on our existing WN 1.7 tags and frameset tags of the Senseval2 verbs in the Penn TreeBank, 95% of the verb instances map directly from sense groups to framesets, with each frameset typically corresponding to two or more sense groups, as illustrated by the tables 1-4 for the verbs 'serve', 'leave', 'pull', and  tactic alternations (such as transitive/inchoative, unspecified object deletion, etc) are analyzed as one sense group. However, in some cases, as illustrated by the verb leave, intransitive and transitive uses are distinguished as different sense groups:  The DEPART sense of the verb can be used transitively if the object specifies the place of departure. The LEAVE BEHIND sense is more general and allows syntactic variation as well as different semantic types of NPs. In Prop-Bank, these groups are unified as one frameset (Frameset  2. Optional Arguments. In PropBank verbs of manner  of motion and verbs of directed motion are usually grouped into one frameset. For example, one of the framesets of the verb pull (TRY) TO CAUSE MOTION unifies the following two group senses:</Paragraph>
      <Paragraph position="8"> Although the frame for the frameset 1 of the verb pull has a 'direction' argument, this argument does not have to be present (or implied), and verbs with this frame can also be understood as verbs of manner of motion in PropBank.</Paragraph>
      <Paragraph position="9"> 3) Syntactic variation of arguments. Syntactic variation in objects can also be used to distinguish sense groups, but are not taken into consideration for distinguishing framesets. Here both noun phrases and sen- null tential complements are contained in the same frameset. These could also be distinguished by the type of event, a physical perception vs. an abstract or mental perception, but these would also not distinguished by  Most of the criteria which are used to split Framesets into groupings, as the tables above illustrate, are semantic. These distinctions, although more fine-grained than Framesets, are still more easily distinguished than WordNet senses.</Paragraph>
      <Paragraph position="10"> Mismatches between Framesets and groupings usually occur for the following two reasons. First, some senses can be missing in the PropBank, if they do not occur in the corpus. Second, given that PropBank is an annotation of the Wall Street Journal, it often distinguishes obscure financial senses of the verb as separate senses.</Paragraph>
    </Section>
  </Section>
  <Section position="4" start_page="0" end_page="0" type="metho">
    <SectionTitle>
4 Experiments with Automatic WSD
</SectionTitle>
    <Paragraph position="0"> We have also been investigating the suitability of these distinctions for training automatic Word Sense Disambiguation systems. The system that we used to tag verbs with their frameset is the same maximum entropy system as that of Dang and Palmer (2002), including both topical and local features. Topical features looked for the presence of keywords occurring anywhere in the sentence and any surrounding sentences provided as context (usually one or two sentences). The set of keywords is specific to each lemma to be disambiguated, and is determined automatically from training data so as to minimize the entropy of the probability of the senses conditioned on the keyword.</Paragraph>
    <Paragraph position="1"> The local features for a verb w in a particular sentence tend to look only within the smallest clause containing w. They include collocational features requiring no linguistic prepro essing beyond part-of-speech tagging (1), syntactic features that capture relations between the verb and its complements (2-4), and semantic features that incorporate information about noun classes for objects (5-6): 1) the word w, the part of speech of w, and words at positions -2, -1, +1, +2, relative to w 2) whether or not the sentence is passive 3) whether there is a subject, direct object, indirect object, or clausal complement (a complement whose node label is S in the parse tree) 4) the words (if any) in the positions of subject, direct object, indirect object, particle, prepositional complement (and its object) 5) a Named Entity tag (PERSON, ORGANIZATION, LOCATION) for proper nouns appearing in (4).</Paragraph>
    <Paragraph position="2"> 6) all possible WordNet synsets and hypernyms for the nouns appearing in (4).</Paragraph>
    <Paragraph position="3"> The system performed well on the English verbs in Senseval-2, achieving an accuracy of 60.2% when tagging verbs with their fine-grained WordNet senses, and 70.2% when tagging with the more coarse-grained sense groups.</Paragraph>
    <Paragraph position="4">  For frameset tagging, we collected a total of 3590 instances of 20 verbs in the PropBank corpus that had been annotated with their framesets. The verbs all had more than one possible frameset and were a subset of the ones used for the English lexical sample task of Senseval-2. Local features for frameset taging were extracted using the gold-standard part-of-speech tags and bracketing of the Penn Treebank. Table 5 shows the number of framesets, the number of instances, and the system accuracy for each verb using 10-fold crossvalidation. The overall accuracy of our automatic frameset tagging was 90.0%, compared to a baseline accuracy of 73.5% if verbs are tagged with their most frequent frameset. While the data is only a subset of that used in Senseval-2, it is clear that framesets can be much more reliably tagged than fine-grained WordNet senses and even sense groups.</Paragraph>
    <Paragraph position="5"> Conclusion This paper described an hierarchical approach to WordNet sense distinctions that provided different types of automatic Word Sense Disambiguation (WSD) systems, which perform at varying levels of accuracy. We have described three different levels of sense granularity, with PropBank Framesets being the most syntactic, the most coarse-grained, and most easily reproduced. A set of manual groupings devised for Senseval2 provides a middle level of granularity that mediates between Framesets and WordNet. For tasks where fine-grained sense distinctions may not be essential such as an AskJeeves information retrieval task, an accurate coarse-grained WSD system such as our Frameset tagger may be sufficient. There is evidence, however, that machine translation of languages as diverse as Chinese and English might require all of the fine-grained sense distinctions of WordNet, and even more (Ng, et al 2003, Palmer, et. al., to appear).</Paragraph>
  </Section>
class="xml-element"></Paper>