<?xml version="1.0" standalone="yes"?>
<Paper uid="H01-1068">
  <Title>A Three-Tiered Evaluation Approach for Interactive Spoken Dialogue Systems</Title>
  <Section position="3" start_page="0" end_page="0" type="intro">
    <SectionTitle>
1. INTRODUCTION
</SectionTitle>
    <Paragraph position="0"> Evaluation of spoken language systems is complicated by the need to balance distinct goals. For collaboration with others in the speech technology community, metrics must be generic enough for comparison to analogous systems. For project management and business purposes, metrics must be specific enough to demonstrate end-user utility and improvement over other approaches to a problem.</Paragraph>
    <Paragraph position="1"> Since 1998, we have developed a spoken language dialogue technology called Listen-Communicate-Show (LCS) and applied it to demonstration systems for U.S. Marines logistics, U.S. Army test data collection, and commercial travel reservations. Our focus is the transition of spoken dialogue technology to military operations. We support military users in a wide range of tasks under diverse conditions. Therefore, our definition of success for LCS is operational success. It must reflect the real world success of our military users in performing their tasks. In addition, for our systems to be considered successful, they must be widely usable and easy for all users to operate with minimal training. Our evaluation methodology must model these objectives.</Paragraph>
    <Paragraph position="2"> With these goals in mind, we have developed a three-tier metric system for evaluating spoken language system effectiveness. The three tiers measure (1) user satisfaction, (2) system support of mission success and (3) component performance.</Paragraph>
  </Section>
class="xml-element"></Paper>