<?xml version="1.0" standalone="yes"?>
<Paper uid="P98-2129">
  <Title>Evaluating Response Strategies in a Web-Based Spoken Dialogue Agent</Title>
  <Section position="9" start_page="785" end_page="785" type="concl">
    <SectionTitle>
7 Conclusion
</SectionTitle>
    <Paragraph position="0"> We have presented an empirical comparison of literal and cooperative query response strategies in TOOT, illustrating the advantages of combining hypothesis testing and PARADISE. By using hypothesis testing to examine how a set of evaluation measures differ as a function of response strategy and task, we show that TOOT's cooperative and literal responses can both lead to greater task success, likelihood of future use, and user need for help, depending on task. By using PARADISE to derive a performance function, we show that a combination of strategy-dependent (perceived task success) and strategy-independent (number of barge-ins, mean recognition score) evaluation measures best predicts overall TOOT performance. Our results elaborate the conditions under which TOOT' s response strategies lead to greater performance, and allow us to make predictions. For example, our performance equation predicts that improving mean recognition and/or judiciously restricting the use of barge-in will enhance performance. Our current research is aimed at automatically adapting dialogue behavior in TOOT, to increase mean recognition and thus overall agent performance (Walker et al., 1998).</Paragraph>
    <Paragraph position="1"> Future work utilizing PARADISE will attempt to generalize our results, to make a more predictive model of agent performance. Performance function estimation needs to be done iteratively over different tasks and dialogue strategies. We plan to evaluate additional cooperative response strategies in TOOT (e.g., intensional summaries (Kalita et al., 1986), summarization and constraint elicitation in isolation), and to combine TOOT data with data from other agents (Walker et al., 1998).</Paragraph>
  </Section>
class="xml-element"></Paper>