<?xml version="1.0" standalone="yes"?>
<Paper uid="W02-0207">
  <Title>Annotating Semantic Consistency of Speech Recognition Hypotheses</Title>
  <Section position="6" start_page="7" end_page="7" type="concl">
    <SectionTitle>
6 Concluding Remarks
</SectionTitle>
    <Paragraph position="0"> In this work we raised the question whether it is possible to reliably annotate speech recognition hypotheses with information about semantic consistency and domain specificity. The motivation for that was to find out whether it is feasible to develop and evaluate a computer program addressing the same task and implementing the algorithm reflected in the annotation scheme.</Paragraph>
    <Paragraph position="1"> We found that humans principally had problems in looking solely at the conceptualized speech recognition hypotheses. This, however, should not be a problem for a machine where the word-to-concept mapping is done automatically and all so-called function words are discarded. In the future it would be interesting to have humans annotate not speech recognition hypotheses per se, but only their automatically generated conceptual mappings.</Paragraph>
    <Paragraph position="2"> Another finding was that the originally proposed annotation scheme does not allow for a high level of agreement between human annotators with respect to semantic consistency.</Paragraph>
    <Paragraph position="3"> Eliminating the class semi-consistent led us, however, to a considerably better reliability of annotations.</Paragraph>
    <Paragraph position="4"> We consider this study as a first attempt to show the feasibility of determining semantic consistency of the output of the speech recognizer. We plan to integrate the results into the domain modeling component and conduct further experiments on semantic consistency and domain detection.</Paragraph>
  </Section>
class="xml-element"></Paper>