<?xml version="1.0" standalone="yes"?>
<Paper uid="H92-1005">
  <Title>Experiments in Evaluating Interactive Spoken Language Systems 1</Title>
  <Section position="2" start_page="0" end_page="28" type="intro">
    <SectionTitle>
INTRODUCTION
</SectionTitle>
    <Paragraph position="0"> For the first two years of the DARPA Spoken Language Program, common evaluation in the ATIS domain has been performed solely with the Common Answer Specification (CAS) protocol \[4\], whereby a system's performance is determined by comparing its output, expressed as a set of database tuples, with one or more predetermined reference answers \[1\]. The CAS protocol has the advantage that system evaluation can be carried out automatically, once the principles for generating the reference answers have been established and a corpus has been annotated accordingly. Since direct comparison across systems can be performed relatively easily with this procedure, we have been able to achieve cross fertilization of research ideas, leading to rapid research progress.</Paragraph>
  </Section>
class="xml-element"></Paper>