<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-0803">
  <Title>SENSEVAL-3 TASK Automatic Labeling of Semantic Roles</Title>
  <Section position="2" start_page="0" end_page="0" type="intro">
    <SectionTitle>
Abstract
</SectionTitle>
    <Paragraph position="0"> The SENSEVAL-3 task to perform automatic labeling of semantic roles was designed to encourage research into and use of the FrameNet dataset. The task was based on the considerable expansion of the FrameNet data since the baseline study of automatic labeling of semantic roles by Gildea and Jurafsky. The FrameNet data provide an extensive body of &amp;quot;gold standard&amp;quot; data that can be used in lexical semantics research, as the basis for its further exploitation in NLP applications. Eight teams participated in the task, with a total of 20 runs. Discussions among participants during development of the task and the scoring of their runs contributed to a successful task. Participants used a wide variety of techniques, investigating many aspects of the FrameNet data. They achieved results showing considerable improvements from Gildea and Jurafsky's baseline study. Importantly, their efforts have contributed considerably to making the complex FrameNet dataset more accessible.</Paragraph>
    <Paragraph position="1"> They have amply demonstrated that FrameNet is a substantial lexical resource that will permit extensive further research and exploitation in NLP applications in the future.</Paragraph>
    <Paragraph position="2"> Introduction Word-sense disambiguation has frequently been criticized as a task in search of a reason. Since a considerable portion of a sense inventory has only a single sense, the question has been raised whether the amount of effort required by disambiguation is worthwhile. Heretofore, the focus of disambiguation has been on the sense inventory and has not examined the major reason why we would have lexical knowledge bases: how the meanings would be represented and thus, available for use in natural language processing applications. At the present time, a major paradigm for representing meaning has emerged in frame semantics, specifically in the FrameNet project.</Paragraph>
    <Paragraph position="3"> A worthy objective for the Senseval community is the development of a wide range of methods for automating frame semantics, specifically identifying and labeling semantic roles in sentences. An important baseline study of this process has recently appeared in the literature (Gildea and Jurafsky, 2002). The FrameNet project (Johnson et al., 2003) has put together a body of hand-labeled data and the Gildea and Jurafsky study has put together a set of suitable metrics for evaluating the performance of an automatic system.</Paragraph>
  </Section>
class="xml-element"></Paper>