<?xml version="1.0" standalone="yes"?>
<Paper uid="W05-1007">
  <Title>Frame Semantic Enhancement of Lexical-Semantic Resources</Title>
  <Section position="5" start_page="62" end_page="63" type="evalu">
    <SectionTitle>
4 Evaluation
</SectionTitle>
    <Paragraph position="0"> Three student judges evaluated SemFrame's results, with 200 frames each assessed by two judges, and 1234 frames each assessed by one judge.</Paragraph>
    <Paragraph position="1"> In evaluating a frame, judges began by examining the set of verb synsets deemed to evoke a common frame and identified from among them the largest subset of synsets they considered to evoke the same frame. This frame--designated the 'target frame'-was simply a mental construct in the judge's mind. For only 9% of the frame judgments were the judges unable to identify a target frame.</Paragraph>
    <Paragraph position="2"> If a target frame was discerned, judges were then asked to evaluate whether the WordNet verb synsets and LDOCE verb senses listed by SemFrame could be used to communicate about the frame the judge had in mind. This evaluation step applied to 6147 WordNet verb synsets and 7148 LDOCE verb senses; in the judges' views, 78% of the synsets and 68% of the verb senses evoke the target frame.</Paragraph>
    <Paragraph position="3"> Judges were asked how well the frame names generated by SemFrame capture the overall target frame. Some 53% of the names were perceived to be satisfactory (good or excellent), with another 25% of the names in the right hierarchy. Only 11% of the names were deemed to be only mediocre and 9% to be unrelated.</Paragraph>
    <Paragraph position="4"> Judges were also asked how well the frame element names generated by SemFrame named a participant or attribute of the target frame. Here 46% of the names were found satisfactory, with another 18% of the names consistent with a target frame participant, but either too general or too narrow. Another 5% of the names were regarded as mediocre and 30% as unrelated.</Paragraph>
    <Paragraph position="5"> Lastly, judges were asked to look for correspondences between target frames and FrameNet frames. While only 17% of the target frames were considered equivalent to a FrameNet frame, many were judged to be hierarchically related; 51% of the FrameNet frames were judged more general than the corresponding SemFrame frame, while 8% were judged more specific. This reflects the need to combine some number of SemFrame frames.</Paragraph>
    <Paragraph position="6"> For 23% of the SemFrame frames, even the best FrameNet match was considered only mediocre.</Paragraph>
    <Paragraph position="7"> These may represent viable frames not yet recognized by FrameNet. Judges also found 3668 verbs in SemFrame that could be appropriately listed for a corresponding frame in FrameNet, but were not.</Paragraph>
    <Paragraph position="8"> These results reveal SemFrame's strengths in in- null ducing frames by enumerating sets of verbs that evoke a shared frame and in naming such frames.</Paragraph>
    <Paragraph position="9"> SemFrame's ability to postulate names for the elements of a frame is less robust, although results in this area are still noteworthy.</Paragraph>
  </Section>
class="xml-element"></Paper>