<?xml version="1.0" standalone="yes"?>
<Paper uid="H01-1068">
  <Title>A Three-Tiered Evaluation Approach for Interactive Spoken Dialogue Systems</Title>
  <Section position="9" start_page="0" end_page="0" type="concl">
    <SectionTitle>
7. CONCLUSION
</SectionTitle>
    <Paragraph position="0"> We have found that structuring evaluation according to the three tiers described above improves the selection of metrics and interpretation of results. While the essence of our approach is domain independent, it does guide the adaptation of metrics to specific applications. First, the three tiers impose a structure that selects certain metrics to constitute a broad pragmatic assessment with minimal data, refining the subject of evaluation. Second, the three tiers organize metrics so that user satisfaction and mission metrics have clear normative semantics (results interpreted as good/bad) and they reveal the impact of low-level metrics (results tied to particular components which may be faulted/lauded). Finally, improvements in selection and interpretation balance satisfaction, effectiveness, and performance, thus imbuing the evaluation process with focus toward utility for practical applications of spoken language dialogue.</Paragraph>
  </Section>
class="xml-element"></Paper>