<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-0803">
  <Title>SENSEVAL-3 TASK Automatic Labeling of Semantic Roles</Title>
  <Section position="5" start_page="0" end_page="0" type="concl">
    <SectionTitle>
3 Discussion
</SectionTitle>
    <Paragraph position="0"> Overall, the results achieved in this SENSEVAL-3 task were quite high. Several teams achieved results much better than those obtained by Gildea and Jurafsky. The average precision of 0.80 for all runs in the unrestricted case is only slightly lower than the 82% accuracy achieved in that study when using presegmented constituents. Many teams achieved precision at or above 0.90, indicating that their routines for classifying constituents is quite good. In view of the fact that the number of frames and frame elements in FrameNet has expanded considerably since the Gildea and Jurafsky study, it appears that the methods employed have become quite accurate in classifying constituents.5 Results for the restricted were also quite good in comparison with the Gildea and Jurafsky study, which achieved 65% precision and 61% recall at the &amp;quot;more difficult task of simultaneously segmenting constituents and identifying their semantic role.&amp;quot; In this task, four teams achieved results between 80 and 90 percent for precision and between 65 and 78 percent for recall.</Paragraph>
    <Paragraph position="1"> The participants in this task used a wide variety of methods and data in their systems. In addition, they used the FrameNet dataset from a wide diversity of perspectives. In some cases, they developed mechanisms for grouping the FrameNet data by part of speech or making use of the nascent inheritance hierarchy in FrameNet. In some cases, they used all frames as a basis for training and in others, they employed novel grouping methods based on the similarities among different frames.</Paragraph>
    <Paragraph position="2"> The successes of many teams seems to indicate that the FrameNet dataset is an excellent lexical resource and that the resources devoted to its development have been quite valuable. The collective efforts of the participants have contributed greatly to making this complex database more accessible and more amenable to even further development, not only for research purposes, but also for use in many NLP applications.</Paragraph>
  </Section>
class="xml-element"></Paper>