<?xml version="1.0" standalone="yes"?>
<Paper uid="C00-2151">
  <Title>An Experiment On Incremental Analysis Using Robust Parsing Techniques</Title>
  <Section position="7" start_page="1027" end_page="1029" type="evalu">
    <SectionTitle>
5 Results
</SectionTitle>
    <Paragraph position="0"> In order to evaluate the potential of the heuristics described above, we have conducted a series of experiments using a grammar that was designed for non-incremental, robust parsing. We tested the increlnental against a non-incremental parser using 222 utterances taken from the VERBMOBIL domain (Wahlster, 1993).</Paragraph>
    <Paragraph position="1">  to tile following criteria: 1 Accuracy. The accuracy (gray bar) describes how many edges of the solutions are correct.</Paragraph>
    <Paragraph position="2"> correct edges accuracy = ~ edges found tNote that the heuristics provide at most one solution and may fail to find any solution.</Paragraph>
    <Paragraph position="3">  Weak recall. We base our recall measure - given as the black bar - on the number of solutions found non-incrementally (which is less than 100%) because we want focus on tile impact of our heuristics, not the coverage of tile grammar. weak recall = @ correct edges @ edges fouud non-incrementally Relative run-time. The run-time required by the incremental procedure as a percentage of the time required by the non-iucremental search algorithm is given as the white bar.</Paragraph>
    <Paragraph position="4"> The difference between tile gray and the black bar is due to errors of the heuristic method, i. e., either because of its incal)ability to find the correct subordination or due to excessive resource demands (which lead to process abortion).</Paragraph>
    <Paragraph position="5">  incremental algorithm to complete while maintaining a relative high degree of quality. Second, the more elaborate the heuristics are, the longer they need to run (as expected) and the better are the results for the accuracy measure. However, the lmuristics D and E could not complete to parse all sentences because in some cases a pre-defined time limit was exceeded; this leads to the observed decrease in weak recall when compared to heuristics C. As expected, a trade-off between computing time and quality can be found. Overall, heuristics C seems to be a good choice because it achieves all accuracy of up to 93.7% in only one fifth of the run-time.</Paragraph>
    <Paragraph position="6">  ~ Time for incremental compared to non-incremental method D Absolute time for incremental (16...20 set to 100%) * The gray bar presents tile normalized time with the time for sentence length between 16 and 20 set to 100%.</Paragraph>
    <Paragraph position="7"> Tile results show that the speedup observed in Figure 2 is not evenly distributed. While the incremental analysis of the short sentences takes longer (2.5 times slower) than the non-incremental algorithm, the opposite is true for longer sentences (10 times faster). However, this is welcome behavior: The incremental procedure takes longer only in those cases that are solved very fast anyway; tim problematic cases are parsed more quickly. This behavior is a first hint that the incremental analysis with re-use of partial results is a step that alleviates the combinatorial explosion of resource demands.</Paragraph>
    <Paragraph position="8">  tile same meaning as ill Figure 2) Finally, Figure 4 compares the quality resulting from heuristics C for different sentence lengths. It turns out that, although a slight decrease is observable, the accuracy is relatively independent of sentence length.</Paragraph>
    <Paragraph position="10"> m 6 ~ An apl)roach to the incremental parsing of natural language utterances has been presented, which is based on tlle idea to use robust parsing techniques to deal with incomplete sentences. It determines a structural description for arbitrary sentence prefixes by searching for the optimal combination of local hypotheses. This search is conducted in a problem space which is repeatedly narrowed down according to the optimal solution found in tile preceding step of analysis.</Paragraph>
    <Paragraph position="11">  The results available so far confirm the initial expectation that the grammar used is robust enough to reliably carry out such a prefix analysis, although it has originally been developed for the non-incremental case. The optimal structure as determined by the parser obviously contains relevant information about the sentence prefix, so that even very simple and cheap heuristics can achieve a considerable level of accuracy. Therefore, large parts of the search space can be excluded fi'oln repeated reanalysis, which eventually makes it even faster than its non-incremental counterpart. Most importantly, the observed speedup grows with the length of the utterance.</Paragraph>
    <Paragraph position="12"> On the other hand, none of the used structure-based heuristics produces a significant iml)rovement of quality even if a large amount of computational resources is spent. Quite a number of cases can be identified where even the most expensive of our heuristics is not strong enough, e. g., the German sentence with a topicalized direct object: DieNOM,ACe Frau sieht derNOM Mama.</Paragraph>
    <Paragraph position="13"> The woman sees tile man.</Paragraph>
    <Paragraph position="14"> The woman, the man sees.</Paragraph>
    <Paragraph position="15"> Here, when analysing the subsentence die Frau sieht, the parser will wrongly consider die Frau as the subject, because it appears to have the right case and there is a clear preference to do so. Later, when the next word comes in, there is no way to allow for dic Frau to change its structural interpretation, because this is not licensed by any of the given heuristics.</Paragraph>
    <Paragraph position="16"> Therefore, substantially more i)roblenl-oriellted heuristics are required, which should take into account not only the ol)timal structure, but also the conflicts caused by it. Using a weak but cheap heuristics, a fast al)proximation of the optimal structure can be obtained within a very restricted search space, and then refined by subsequent structural transformations (Foth et al., 2000). To a certain degree this resembles the idea of applying reason maintenance techniques for conflict resolution in incremental parsing (Wir6n, 1990). In deciding which strategy is good enough to find the necessary first approximation the results of this paper might play a crucial role, since the I)ossible contribution of individual heuristics in such all extended fi'amework can be precisely estimated.</Paragraph>
  </Section>
class="xml-element"></Paper>