<?xml version="1.0" standalone="yes"?>
<Paper uid="W97-0603">
  <Title>GENERALITY AND OBJECTIVITY Central Issues in Putting a Dialogue Evaluation Tool into Practical Use</Title>
  <Section position="8" start_page="24" end_page="24" type="concl">
    <SectionTitle>
6. Conclusion and Future Work
</SectionTitle>
    <Paragraph position="0"> We find the results reported in this paper encouraging.</Paragraph>
    <Paragraph position="1"> The tool has generalised well to the Sundial corpus and some amount of objectivity has been demonstrated with respect to type identification and classification. As this was out first attempt at using the tool independently of one another, we intend to repeat the exercise using the insights gained. Two times 15 Sundial dialogues will be used for the purpose. Following that, we plan to repeat the experiment with a small sub-corpus of the Philips corpus which comprises 13.500 field test dialogues conceming train timetable information (Aust et al. 1995).</Paragraph>
    <Paragraph position="2"> This will add a new dialogue and task type, as well as the new circumstances of a field trial to the generality test of the tool. If and when convincing generality and a satisfactory degree of objectivity in using DET have been achieved, a final transfer problem must be addressed. This problem concerns how to transfer DET to other developers in some &amp;quot;packaged&amp;quot; form which does not assume person-to-person tuition. This should enable other SLDS developers to quickly and efficiently learn to use DET at the same level of objectivity as has been achieved during the tests of the tool. Only then will DET be ready for inclusion among the growing number of dialogue engineering best practice development and evaluation tools. As a first step in addressing the transfer problem, we have recently included a DET novice in the team. He is an experienced computational linguist but with little experience in SLDS development. We are investigating what it takes to make him an expert in using DET by having him analyse the same Sundial sub-corpus as was reported on above and we hope that he will participate in the planned second Sundial sub-corpus exercise. Following these steps, the final task will be to define an explicit and simple training scheme for how to become an expert in using the tool.</Paragraph>
  </Section>
class="xml-element"></Paper>