<?xml version="1.0" standalone="yes"?>
<Paper uid="H94-1114">
  <Title>ANNOTATION OF ATIS DATA</Title>
  <Section position="1" start_page="0" end_page="0" type="metho">
    <SectionTitle>
ANNOTATION OF ATIS DATA
</SectionTitle>
    <Paragraph position="0"/>
  </Section>
  <Section position="2" start_page="0" end_page="0" type="metho">
    <SectionTitle>
PROJECT GOALS
</SectionTitle>
    <Paragraph position="0"> The performance of spoken lartguage systems on utterances from the ATIS domain is evaluated by comparing system-produced responses with hand-crafted (and -verified) standard responses to the same utterances. The objective of SRI's annotation project is to provide SLS system developers with the range of correct responses to human utterances produced during experimental sessions with ATIS domain interactive systems. These correct responses are then used in system training and evaluation.</Paragraph>
  </Section>
  <Section position="3" start_page="0" end_page="0" type="metho">
    <SectionTitle>
PLANS FOR THE COMING YEAR
</SectionTitle>
    <Paragraph position="0"> In the next year, SKI will continue to provide MADCOW annotation and other services to the DARPA community. In addition to tasks previously performed, SRI will assume a greater role in the resolution of training data bug reports.</Paragraph>
  </Section>
  <Section position="4" start_page="0" end_page="471" type="metho">
    <SectionTitle>
RECENT RESULTS
</SectionTitle>
    <Paragraph position="0"> Since June 1991. SRI has produced classification and response files for about 13,000 utterances of training data (4000 of these since January, 1993). A dry run system evaluation and three official evaluations have been held since the project began in 1991. SKI has produced the standard responses for all of these evaluations; in all, about 3300 ntteranc~s.</Paragraph>
    <Paragraph position="1"> These tests were performed according to the Common Answer Specification (CAS) protocol which is used in training. All systems are evaluated on a common set of data, with system responses measured against otiicial reference answers produced at SRI in the same manner as the training data.</Paragraph>
    <Paragraph position="2"> In addition to producing the classification and standard response files, SKI takes an active role in the adjudication of test and training data bug reports, initiates nearly all of the changes to the Principles oflnterpretation document (a basic set of principles for interpreting the meaning of ATIS sentences agreed upon by the DARPA community), and continues to support NIST by modifying software and acthag as a consultant regarding the correct annotation of data.</Paragraph>
  </Section>
  <Section position="5" start_page="471" end_page="471" type="metho">
    <SectionTitle>
J
</SectionTitle>
    <Paragraph position="0"/>
  </Section>
class="xml-element"></Paper>