<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-2315">
  <Title>Speech Graffiti habitability: What do users really say?</Title>
  <Section position="4" start_page="0" end_page="0" type="metho">
    <SectionTitle>
2 Method
</SectionTitle>
    <Paragraph position="0"> Our data was generated from a user study in which participants were asked to complete tasks using both a Speech Graffiti interface to a telephone-based movie information system (MovieLine) and a natural language interface to the same data. Tasks were designed to have the participants explore a variety of the functions of the systems (e.g. &amp;quot;list what's playing at the Squirrel Hill Theater&amp;quot; and &amp;quot;find out &amp; write down what the ratings are for the movies showing at the Oaks Theater&amp;quot;).</Paragraph>
    <Paragraph position="1"> After interacting with each system, each participant completed a user satisfaction questionnaire rating 34 subjective-response items on a 7-point Likert scale. This questionnaire was based on the Subjective Assessment  faction for Speech Graffiti MovieLine.</Paragraph>
    <Paragraph position="2"> of Speech System Interfaces (SASSI) project (Hone &amp; Graham, 2001) and included statements such as &amp;quot;I always knew what to say to the system&amp;quot; and &amp;quot;the system makes few errors.&amp;quot; An overall user satisfaction rating was calculated for each user by averaging that user's scores for each of the 34 response items. Users were also asked a few comparison questions, including system preference. In this analysis we were only concerned with results from the Speech Graffiti MovieLine interactions and not the natural language MovieLine interactions (see Tomko &amp; Rosenfeld, 2004). System presentation order was balanced and had no significant effect on grammaticality measures.</Paragraph>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
2.1 Participants
</SectionTitle>
      <Paragraph position="0"> Twenty-three participants (12 female, 11 male) accessed the systems via telephone in our lab. Most were undergraduate students from Carnegie Mellon University and all were native speakers of American English. We also asked users whether they considered themselves &amp;quot;computer science or engineering people&amp;quot; (CSE) and how often they did computer programming; the distributions of these categories were roughly equal.</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
2.2 Training
</SectionTitle>
      <Paragraph position="0"> The Speech Graffiti approach requires users to learn the system prior to using it via a brief tutorial session. 15 participants received unsupervised Speech Graffiti training consisting of a self-directed, web-based tutorial that presented sample dialog excerpts (in text) and proposed example tasks to the user. The other eight participants received supervised Speech Graffiti training. This training used the same web-based foundation as the unsupervised version, but participants were encouraged to ask the experimenter questions if they were unsure of anything during the training session.</Paragraph>
      <Paragraph position="1"> Both supervised and unsupervised training sessions were balanced between web-based tutorials that used examples from the MovieLine and from a FlightLine system that provided simulated flight arrival, departure, and gate information. This enabled us to make an initial assessment of the effects of in-domain training.</Paragraph>
    </Section>
    <Section position="3" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
2.3 Analysis
</SectionTitle>
      <Paragraph position="0"> The user study generated 4062 Speech Graffiti MovieLine utterances, where an utterance is defined as one chunk of speech input sent to our Sphinx II speech recognizer (Huang et al., 1993). We removed all utterances containing non-task-related or unintelligible speech, or excessive noise or feed, resulting in a cleaned set of 3626 utterances (89% of the total). We defined an utterance to be grammatical if the Phoenix parser (Ward, 1990) used by the system returns a complete parse with no extraneous words.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>