<?xml version="1.0" standalone="yes"?>
<Paper uid="W00-1016">
  <Title>Dialogue Helpsystem based on Flexible Matching of User Query with Natural Language Knowledge Base</Title>
  <Section position="8" start_page="147" end_page="148" type="evalu">
    <SectionTitle>
7 Evaluation
</SectionTitle>
    <Paragraph position="0"> The helpsystem started its service on July 1999 as part of the CIMS web service. All conversation logs between users and the system have been stored as a dialogue database.</Paragraph>
    <Paragraph position="1"> In the dialogue database, each dialogue between a user and the system is segmented into task units manually. We call this unit a session. Figure 4 shows the number of sessions and their evaluation of each week from July 5th to January 30th. On average, there are 70 sessions in a week; a dialogue with a user and the system consists of 2.1 sessions, which means a user asks 2.1 topics in one dialogue; one session consists of 3.2 turns.</Paragraph>
    <Paragraph position="2"> The evaluation of sessions is based on the following criteria.</Paragraph>
    <Paragraph position="3"> Success: The system could return a satisfactory answer.</Paragraph>
    <Paragraph position="4"> Failure:Input Analyzer: 'The system could not response properly because of the input analysis error, mostly the lack of utterance-pattern rules. Utterance~ pattern rules are added whenever the lack is found.</Paragraph>
    <Paragraph position="5"> Failure:Dialog Manager: The system could not response properly because of the dialogue manager error. Dialogue manager error comes both from simple bugs of the system and from unnoticed patterns of the user response.</Paragraph>
    <Paragraph position="6"> For example, when the system asks &amp;quot;select from A and B&amp;quot; expecting the answer &amp;quot;A&amp;quot; or &amp;quot;B&amp;quot;, a user might answer &amp;quot;the latter&amp;quot;. The system is modified whenever necessary.</Paragraph>
    <Paragraph position="7"> Failure:Knowledge: The system could not answer the question because of the lack of knowledge. This is the major reason of the failure as shown in Figure 4. Though the knowledge base is being extended step by step, the range of the user  query is unlimited, including troubles in using PCs and advanced settings of software/hardware. null Failure:Difilcult: Current system architecture could not handle the question. For example, a user sometimes asks &amp;quot;what is the difference between A and B&amp;quot;, or when the system asks &amp;quot;select from A and B&amp;quot;, a user answers &amp;quot;I don't know&amp;quot;. In order to handle such utterances, we are planning to improve the system to exploit definitions of &amp;quot;A&amp;quot; and &amp;quot;B&amp;quot;. Out of scope: Out of the system domain, such as questions about telephone charges in using PPP or the Y2K problem. null Miscellaneous: Such as &amp;quot;hello&amp;quot;, &amp;quot;this is a test&amp;quot; or just a simple typo like &amp;quot;a&amp;quot;. The success ratio, that is, the ratio of Success over Success plus Failure, of the whole period is 37%. The system became stable around October 1999, and the success ratio after that (14 weeks) is 39%. Considering relatively wide domain the system have to cover, we feel the success ratio is reasonable, and the system is contributing to CIMS to some extent by handling simple FAQs like &amp;quot;how to change my password&amp;quot;.</Paragraph>
  </Section>
class="xml-element"></Paper>