<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-2315">
  <Title>Speech Graffiti habitability: What do users really say?</Title>
  <Section position="6" start_page="0" end_page="0" type="concl">
    <SectionTitle>
4 Discussion
</SectionTitle>
    <Paragraph position="0"> We have shown a significant correlation between grammaticality and user satisfaction for the Speech Graffiti system. Grammaticality scores were generally high and tended to increase over time, demonstrating that the system is acceptably habitable.</Paragraph>
    <Paragraph position="1"> Based on the data shown in Fig.1, it appears that 80% is a good target for Speech Graffiti grammaticality. Nearly all participants with grammaticality scores over 80% gave positive (i.e. &gt; 4) user satisfaction scores, and more than half of our users achieved this level. Furthermore, users with grammaticality above 80% completed an average of 6.9 tasks, while users with grammaticality below 80% completed an average of only 3.5 tasks. A fundamental question for our future work is &amp;quot;what can we do to help everyone speak within the bounds of the system at the 80% level?&amp;quot; Several possible refinements are immediately apparent beyond fixing our trivial grammar problems. System responses to options should be reworked to reduce incorrect lexical entrainment and alleviate slot-only deviations. The out-of-vocabulary instances can be analyzed to decide whether certain synonyms should be added to the current system, although this will generate only domain-specific improvements. Many ungrammaticality types can also be addressed through refinements to Speech Graffiti's help and tutorial functions.</Paragraph>
    <Paragraph position="2"> Addressing the general NL syntax category poses the biggest problem. Although it is responsible for the largest portion of ungrammaticalities, simply changing the grammar to accommodate these variations would likely lead to increased system complexity. A main concern of our work is domain portability, and Speech Graffiti's standard structure currently allows for fairly rapid creation of new interfaces (Toth et al., 2002). Any natural language expansion of Speech Graffiti grammar will have to be balanced with the ability to port such a grammar to all domains. We are currently analyzing the ungrammatical utterances in this and the time syntax categories to determine whether any Speech Graffiticonsistent modifications could be made to the interface. However, most of the improvement in this area will likely have to be generated via better help and training.</Paragraph>
    <Paragraph position="3"> An important additional finding from this work is the scope of general NL syntax deviations. Considering items like movie and theater names as equivalence class members, the NL utterances used by participants in the Speech Graffiti system reduced to 94 patterns. In comparison, the NL utterances used by participants in the natural language MovieLine reduced to about 580 patterns. One of the main differences between the NL patterns in the two systems was the lack of conversational phrases like &amp;quot;can you give me...&amp;quot; and &amp;quot;I would like to hear about...&amp;quot; in the Speech Graffiti system. Thus the knowledge that they are interacting with a restricted language system seems to be enough to make users speak more simply, matching results from Ringle &amp; Halstead-Nussloch (1989).</Paragraph>
    <Paragraph position="4"> Although many of our ungrammaticality types may appear to be specific to Speech Graffiti, they reinforce lessons applicable to most speech interfaces. The slot-only issue demonstrates that lexical entrainment truly is a factor in spoken language interfaces and its effects should not be underestimated. Out-of-vocabulary words are a persistent problem, and keywords should be chosen with care to ensure that they are task-appropriate and that their functions are as intuitive as possible.</Paragraph>
    <Paragraph position="5"> Overall, this study has provided us with a target level for Speech Graffiti-grammaticality, suggested changes to the language and provided insight about what aspects of the system might need greater support through help and tutorial functions. We plan to implement changes based on these results and re-evaluate the system through further user testing.</Paragraph>
  </Section>
class="xml-element"></Paper>