<?xml version="1.0" standalone="yes"?>
<Paper uid="P04-1048">
  <Title>Inducing Frame Semantic Verb Classes from WordNet and LDOCE</Title>
  <Section position="8" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
5 Results
</SectionTitle>
    <Paragraph position="0"> We explored a range of thresholds in the final stage of the algorithm. In general, the lower the1 threshold, the looser the verb grouping. The number of verb senses retained (out of 12,663 non-phrasal verb senses in LDOCE) and the verb sense groups produced by using these thresholds are recorded in Table 2.</Paragraph>
  </Section>
  <Section position="9" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
6 Evaluation
</SectionTitle>
    <Paragraph position="0"> One of our goals is to produce sets of verb senses capable of extending FrameNet's coverage while requiring reasonably little post-editing. This goal has two subgoals: identifying new frames and identifying additional lexical units that evoke  previously recognized frames. We use the hand-crafted FrameNet, which is of reliably high precision, as a gold standard for the initial2 evaluation of SemFrame's ability to achieve these subgoals. For the first, we evaluate SemFrame's ability to generate frames that correspond to FrameNet's frames, reasoning that the system must be able to identify a large proportion of known frames if the quality of its output is good enough to identify new frames. (At this stage we do not measure the quality of new frames.) For the second subgoal we can be more concrete: For frames identified by both systems, we measure the degree to which the verbs identified by SemFrame can be shown to evoke those frames, even if FrameNet has not identified them as frame-evoking verbs.</Paragraph>
    <Paragraph position="1"> FrameNet includes hierarchically organized frames of varying levels of generality: Some semantic areas are covered by a general frame, some by a combination of specific frames, and some by a mix of general and specific frames.</Paragraph>
    <Paragraph position="2"> Because of this variation we determined the degree to which SemFrame and FrameNet overlap by automatically finding and comparing corresponding frames instead of fully equivalent frames. Frames correspond if the semantic scope of one frame is included within the semantic For the clustering algorithm used, the clustering FrameNet's frames are more syntactically than1 threshold range is open-ended. The values semantically motivated (e.g., EXPERIENCER-OBJECT, investigated in the evaluation are fairly low. EXPERIENCER-SUBJECT). Certain constraints imposed by FrameNet's2 development strategy restrict its use as a full-fledged gold standard for evaluating semantic frame induction. (1) As of summer 2003, only 382 frames had been identified within the FrameNet project. (2) Low recall affects not only the set of semantic frames identified by FrameNet, but also the sets of frame-evoking units listed for each frame. No verbs are listed for 38.5% of FrameNet's frames, while another 13.1% of them list only 1 or 2 verbs. The comparison here is limited to the 197 FrameNet frames for which at least one verb is listed with a counterpart in LDOCE. (3) Some of a. Relates LDOCE verb senses that are defined in terms of the same verb Input. D, a set of (verb_sense_id, def_verb) pairs, where def_verb = the verb in terms of whichd verb_sense_id is definedd Step 1. forall v that exist as def_verb in D, form DV D, by extracting all (verb_sense_id, def_verb)v pairs where v = def_verb Step 2. remove all DV for which  |DV  |&gt; 40v v Step 3. forall v that exist as def_verb in D, return all combinations of two members from DVv b. Relates LDOCE verb senses that share a common stem Input. D, a set of (verb_sense_id, verb_stem) pairs, where verb_stem = the stem for the verb on whichd  verb_sense_id is basedd Step 1. forall m that exist as verb_stem in D, form DV D, by extracting all (verb_sense_id,m verb_stem) pairs where m = verb_stem Step 2. forall m that exist as verb_stem in D, return all combinations of two members from DVv c. Extracts explicit sense-linking relationships in LDOCE Input. D, a set of (verb_sense_id, def) pairs, where def = the definition for verb_sense_idd d Step 1. forall d D, if def contains compare or opposite note, extract related_verb from note; generated (verb_sense_id , related_verb ) pair d d Step 2. forall d D, if def defines verb_sense_id in terms of a related standalone verb (in BLOCKd d  CAPS), extract related_verb from definition; generate (verb_sense_id , related_verb ) pair d d Step 3. forall (verb_sense_id , related_verb ) pairs, if there is only one sense of related_verb , choose itd d d and return (verb_sense_id , related_verb_sense_id ), else apply generalized mappingd d algorithm to return (verb_sense_id , related_verb_sense_id ) pairs where overlap occurs ind d the glosses of verb_sense_id and related_verb_sense_idd d d. Relates verb senses that share general or specific subject field codes in LDOCE Input. D, a set of (verb_sense_id, subject_code) pairs, where subject_code = any 2- or 4-characterd subject field code assigned to verb_sense_id Step 1. forall c that exist as subject_code in D, form DV D, by extracting all (verb_sense_id,c subject_code) pairs where c = subject_code Step 2. forall c that exist as subject_code in D, return all combinations of two members from DVv e. Extracts (direct or extended) semantic relationships in WordNet Input. WordNet data file for verb synsets Step 1. forall synset lines in input file return (synset, related_synset) pairs for all synsets directly related through hyponymy, antonymy, entailment, or cause_to relationships in WordNet (for extended relationship pairs, also return (synset, related_synset) pairs for all synsets within hyponymy tree, i.e., no matter how many levels removed) f. Relates LDOCE verb senses that map to the same WordNet synset Input. mapping of LDOCE verb senses to WordNet synsets Step 1. forall lines in input file return all combinations of two LDOCE verb senses mapped to the same WordNetlsynset Figure 3. Algorithms for Generating Non-clustering-based Verb Pairs scope of the other frame or if the semantic scopes SemFrame's verb classes list specific LDOCE of the two frames have significant overlap. Since verb senses. In extending FrameNet, verbs from FrameNet lists evoking words, without SemFrame would be word-sense-disambiguated specification of word sense, the comparison was in the same way that FrameNet verbs currently done on the word level rather than on the word are, through the correspondence of lexeme and sense level, as if LDOCE verb senses were not frame.</Paragraph>
    <Paragraph position="3"> specified in SemFrame. However, it is clearly Incompleteness in the listing of evoking verbs specific word senses that evoke frames, and in FrameNet and SemFrame precludes a straight-forward detection of correspondences between incrust, and ornament. Two of the verbs--adorn their frames. Instead, correspondence between and decorate--are shared. In addition, the frame FrameNet and SemFrame frames is established names are semantically related through a using either of two somewhat indirect approaches. WordNet synset consisting of decorate, adorn In the first approach, a SemFrame frame is (which CatVar relates to ADORNING), grace, deemed to correspond to a FrameNet frame if the ornament (which CatVar relates to two frames meet both a minimal-overlap ORNAMENTATION), embellish, and beautify. The criterion (i.e., there is some, perhaps small, two frames are therefore designated as overlap between the FrameNet and SemFrame corresponding frames by meeting both the framesets) and a frame-name-relatedness minimal-overlap and the frame-name relatedness criterion. The minimal-overlap criterion is met if criteria.</Paragraph>
    <Paragraph position="4"> either of two conditions is met: (1) If the In the second approach, a SemFrame frame is FrameNet frame lists four or fewer verbs (true of deemed to correspond to a FrameNet frame if the over one-third of the FrameNet frames that list two frames meet either of two relatively stringent associated verbs), minimal overlap occurs when verb overlap criteria, the majority-match criterion any one verb associated with the FrameNet frame or the majority-related criterion, in which case matches a verb associated with a SemFrame examination of frame names is unnecessary. frame. (2) If the FrameNet frame lists five or The majority-match criterion is met if the set more verbs, minimal overlap occurs when two or of verbs shared by FrameNet and SemFrame more verbs in the FrameNet frame are matched by framesets account for half or more of the verbs in verbs in the SemFrame frame. either frameset. For example, the APPLY_HEAT The looseness of the minimal overlap frame in FrameNet includes 22 verbs: bake, criterion is tightened by also requiring that the blanch, boil, braise, broil, brown, char, coddle, names of the FrameNet and SemFrame frames be cook, fry, grill, microwave, parboil, poach, roast, closely related. Establishing this frame-name saute, scald, simmer, steam, steep, stew, and relatedness involves identifying individual toast, while the BOILING frame in SemFrame components of each frame name and augmenting includes 7 verbs: boil, coddle, jug, parboil,3 this set with morphological variants from CatVar poach, seethe, and simmer. Five of these (Habash and Dorr 2003). The resulting set for verbs--boil, coddle, parboil, poach, and each FrameNet and SemFrame frame name is simmer--are shared across the two frames and then searched in both the noun and verb WordNet constitute over half of the SemFrame frameset. networks to find all the synsets that might Therefore the two frames are deemed to correspond to the frame name. To these sets are correspond by meeting the majority-match also added all synsets directly related to the criterion.</Paragraph>
    <Paragraph position="5"> synsets corresponding to the frame names. If the The majority-related criterion is met if half or resulting set of synsets gathered for a FrameNet more of the verbs from the SemFrame frame are frame name intersects with the set of synsets semantically related to verbs from the FrameNet gathered for a SemFrame frame name, the two frame (that is, if the precision of the SemFrame frame names are deemed to be semantically verb set is at least 0.5). To evaluate this criterion, related. each FrameNet and SemFrame verb is associated For example, the FrameNet ADORNING frame with the WordNet verb synsets it occurs in, contains 17 verbs: adorn, blanket, cloak, coat, augmented by the synsets to which the initial sets cover, deck, decorate, dot, encircle, envelop, of synsets are directly related. If the sets of festoon, fill, film, line, pave, stud, and wreathe. synsets corresponding to two verbs share one or The SemFrame ORNAMENTATION frame contains more synsets, the two verbs are deemed to be 12 verbs: adorn, caparison, decorate, embellish, semantically related. This process is extended embroider, garland, garnish, gild, grace, hang, one further level, such that a SemFrame verb found by this process to be semantically related to a SemFrame verb, whose semantic relationship to a FrameNet verb has already been established, will also be designated a frame-evoking verb. If half or more of the verbs listed for a SemFrame frame are established as evoking the same frame as the list of WordNet verbs, then the FrameNet All SemFrame frame names are nouns. (See3 Green and Dorr, 2004 for an explanation of their selection.) FrameNet frame names (e.g., ABUNDANCE, A C T I V I T Y _ S T A R T , C A U S E _ T O _ B E _ W E T , INCHOATIVE_ATTACHING), however, exhibit considerable variation.</Paragraph>
    <Paragraph position="6"> and SemFrame frames are hypothesized to bound on the task, i.e., 100% recall and 100% correspond through the majority-related criterion. precision. The Lin &amp; Pantel results are here a For example, the FrameNet ABUNDANCE lower bound for automatically induced semantic frame includes 4 verbs: crawl, swarm, teem, and verb classes and probably reflect the limitations of throng. The SemFrame FLOW frame likewise using only corpus data. Among efforts to develop includes 4 verbs: pour, teem, stream, and semantic verb classes, SemFrame's results pullulate. Only one verb--teem--is shared, so correspond more closely to semantic frames than the majority-match criterion is not met, nor is the do others.</Paragraph>
    <Paragraph position="7"> related-frame-name criterion met, as the frame names are not semantically related. The majority-related criterion, however, is met through a WordNet verb synset that includes pour, swarm, stream, teem, and pullulate.</Paragraph>
    <Paragraph position="8"> Of the 197 FrameNet frames that include at least one LDOCE verb, 175 were found to have a corresponding SemFrame frame. But this 88.8% recall level should be balanced against the precision ratio of SemFrame verb framesets.</Paragraph>
    <Paragraph position="9"> After all, we could get 100% recall by listing all verbs in every SemFrame frame.</Paragraph>
    <Paragraph position="10"> The majority-related function computes the precision ratio of the SemFrame frame for each pair of FrameNet and SemFrame frames being compared. By modifying the minimum precision threshold, the balance between recall and precision, as measured using F-score, can be investigated. The best balance for the SemFrame version is based on a clustering threshold of 2.0 and a minimum precision threshold of 0.4, which yields a recall of 83.2% and overall precision of 73.8%.</Paragraph>
    <Paragraph position="11"> To interpret these results meaningfully, one would like to know if SemFrame achieves more FrameNet-like results than do other available verb category data, more specifically the 258 verb classes from Levin, the 357 semantic verb classes of WordNet 1.7.1, or the 272 verb clusters of Lin and Pantel, as described in Section 2.</Paragraph>
    <Paragraph position="12"> For purposes of comparison with FrameNet, Levin's verb class names have been hand-edited to isolate the word that best captures the semantic sense of the class; the name of a WordNet-based frame is taken from the words for the root-level synset; and the name of each Lin and Pantel cluster is taken to be the first verb in the cluster.4 Evaluation results for the best balance between recall and precision (i.e., the maximum F-score) of the four comparisons are summarized in Table 3. FrameNet itself constitutes the upper</Paragraph>
  </Section>
class="xml-element"></Paper>