<?xml version="1.0" standalone="yes"?>
<Paper uid="P03-2027">
  <Title>Dialog Navigator : A Spoken Dialog Q-A System based on Large Text Knowledge Base</Title>
  <Section position="5" start_page="2" end_page="2" type="evalu">
    <SectionTitle>
5 Experimental Evaluation
</SectionTitle>
    <Paragraph position="0"> We evaluated the system performance experimentally. For the experiments, we had 4 subjects, who were accustomed to using computers. They made utterances by following given 10 scenarios and also made several utterances freely. In total, 53 utterances were recorded. Figure 5 shows two successful dialogs by confirmation using confidence in recognition and by that using significance for retrieval. We experimented on the system using the 53 recorded utterances by the following methods:  (1) Using correct transcription of recorded utterance, including fillers.</Paragraph>
    <Paragraph position="1"> (2) Using speech recognition results from which only fillers were removed.</Paragraph>
    <Paragraph position="2"> (3) Using speech recognition results and making confirmation by confidence in recognition.</Paragraph>
    <Paragraph position="3"> (4) Using C6-best candidates of speech recognition and making confirmation by significance for retrieval. Here, C6 BPBF.</Paragraph>
    <Paragraph position="4"> (5) Using C6-best candidates of speech recognition  and both measures in (3) and (4).</Paragraph>
    <Paragraph position="5"> In these experiments, we assumed that users always correctly answer system's asking backs. We regarded a retrieval as a successful one if a relevant text was contained in ten high-scored retrieval texts. Table 2 shows the result. It indicates that our confirmation methods for fixing speech recognition errors improve the success rate. Furthermore, the success rate with both measures gets close to that with the transcriptions. Considering that the speech recognition correctness is about 70%, the proposed dialog strategy is effective.</Paragraph>
  </Section>
class="xml-element"></Paper>