<?xml version="1.0" standalone="yes"?>
<Paper uid="H92-1011">
  <Title>Experiences Collecting Genuine Spoken Enquiries using WOZ Techniques</Title>
  <Section position="6" start_page="61" end_page="63" type="evalu">
    <SectionTitle>
6. RESULTS
</SectionTitle>
    <Paragraph position="0"> The service went 'live' for the first time during November 1991 but, due to the limited advertising campaign and a relatively small population of potential users, the number of calls received (by the end of December 1991) was rather smaller than had been initially hoped for. Nevertheless, the data collected during that period already capture some interesting general features of genuine spoken human-machine interaction.</Paragraph>
    <Paragraph position="1"> For example, with some calls there was considerable background office noise (some callers appeared to be using loud-speaking telephones). Also, callers occasionally chuckled to themselves or made asides to other people in their vicinity (including statements along the lines of &amp;quot;Hey, I'm talking to a machine&amp;quot; and constant references to 'it') - although this confirmed that the callers were convinced by the wizard's voice it also indicated that they believed that the system automatically knew when it was being addressed! Some callers interrupted the wizard, and at least one mimicked the robotic style of the wizard's voice.</Paragraph>
    <Paragraph position="2"> It was also noticeable that, although the human-wizard dialogues were all concerned with planning particular routes, most of the human-human dialogues were about the nature of the service itself. In other words, the users who dealt with the wizard seemed to assume that such a system would not be able to provide explanations about what it could and couldn't do - and so they didn't ask.</Paragraph>
    <Paragraph position="3"> In summary, during the period 13th of November 1991 to 5th of December 1991 the service received a total of twenty-two calls.</Paragraph>
    <Paragraph position="4"> The average length of a wizard operated call was two minutes and approximately three minutes for the human operated calls.</Paragraph>
    <Paragraph position="5"> A preliminary analysis of the transcripts produced a variety of interesting statistics on caller behaviour. In particular it was found that, on average, there were signifi- null cantly fewer words spoken by the caller in each turn of the human-wizard condition than in the human-human condition. Also, although the rate of &amp;quot;uhms&amp;quot; and &amp;quot;errs&amp;quot; was about the same in both conditions, callers seemed to be more polite to the machine than to the human  There were two interesting exceptions to the general statistics and the corresponding data was excluded from the analysis.</Paragraph>
    <Paragraph position="6"> In the first exception, the behaviour of one caller to the wizard was very different to the other human-wizard interactions but very much in line with the human-human dialogues. A check of the corresponding transcripts revealed that the following utterance was spoken by the caller at the start of the call - &amp;quot;Oh, sorry, it sounded  like a machine that was talking&amp;quot; - after which the caller appeared to continue with the assumption that he was talking to a human operator (despite the peculiar voice)! In the second exception, another caller, this time to the human operator, exhibited behaviour which was more in line with the human-wizard dialogues. In this case it transpired that the operator had forgotten to switch off the voice disguise unit at the very beginning of the call. Thus this caller seemed to believe he was talking to a machine even after the operator's voice suddenly returned to normal!</Paragraph>
  </Section>
class="xml-element"></Paper>