<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-3218">
  <Title>Mining Spoken Dialogue Corpora for System Evaluation and Modeling</Title>
  <Section position="2" start_page="0" end_page="0" type="intro">
    <SectionTitle>
1 Introduction
</SectionTitle>
    <Paragraph position="0"> The deployment of large scale automatic spoken dialog systems, like How May I Help You?SM (HMIHY) (Gorin et al., 1997), makes available large corpora of real human-machine dialog interactions. Traditionally, this data is used for supervised system evaluation. For instance, in (Kamm et al., 1999) they propose a static analysis aimed at measuring the performance of a dialog system, especially in an attempt to automatically estimate user satisfaction. In (Hastie et al., 2002), a dynamic strategy in the error handling process is proposed. In all these studies, supervised learning techniques are used in order to classify dialogs to predict user satisfaction or dialog failures.</Paragraph>
    <Paragraph position="1"> A novel approach to the exploitation of dialog corpora is for speech recognition and language understanding modeling. In fact, such corpora allow for a multidimensional analysis of speech and language models of dialog systems. Our work di ers from previous studies in the algorithmic approach and learning scenario.</Paragraph>
    <Paragraph position="2"> First we use unsupervised speech mining techniques. We apply data clustering methods to large spoken dialog corpora. Two kinds of clustering methods are used: a hierarchical one based on decision trees and the optimization of a statistical criterion; a partitional one based on a k-means algorithm applied to vectors representing the dialogs. We interpret the clusters obtained and de ne a label for each of them.</Paragraph>
    <Paragraph position="3"> Second we perform our analyses on large corpora of real dialogs collected from deployed systems. These log les contain a trace of the interaction between the users and a particular system at a certain point in time. Our goal is to highlight the structures behind these traces.</Paragraph>
    <Paragraph position="4"> Lastly, we investigate several ways of encoding a dialog into a multidimensional structure in order to apply data clustering methods. Information about the system channel and the user channel are discussed and two ways of encoding are proposed, one for hierarchical clustering and the other for partitional clustering.</Paragraph>
    <Paragraph position="5"> The clusters obtained can be used to learn about the behavior of the system with regards to the automation rate and the type of interaction (e.g. easy vs di cult dialog). Moreover, the clusters can be used on-the- y to automatically adapt the language model, the understanding module and even the dialog strategy to better t the kind of interaction detected.</Paragraph>
    <Paragraph position="6"> In this study, we present two levels of clustering: clustering at the utterance level and the System: How may I help you? User: Hello? Call-type: Hello System: Hello, how may I help you? User: I have a question.</Paragraph>
    <Paragraph position="7"> Call-type: Ask(Info) System: OK, What is your question? User: I would like to know my account balance.</Paragraph>
  </Section>
class="xml-element"></Paper>