<?xml version="1.0" standalone="yes"?>
<Paper uid="H93-1006">
  <Title>A PORTABLE APPROACH TO LAST RESORT PARSING AND INTERPRETATION</Title>
  <Section position="7" start_page="33" end_page="34" type="evalu">
    <SectionTitle>
5. Results
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="33" end_page="33" type="sub_section">
      <SectionTitle>
5.1. Air Traffic Control
</SectionTitle>
      <Paragraph position="0"> We performed experiments on a set of 233 utterances in the ATC domain, incorporating utterances from two different controllers. One was guiding planes which had just landed; the other was guiding planes as they taxied in preparation for takeoff.</Paragraph>
      <Paragraph position="1"> Substantial benefits are gained from using backup, or &amp;quot;last-resort&amp;quot; processing, after normal parsing fails or a timeout occurs. Figure 1 shows that application accuracy is improved by the use of such processing, at two different settings of the timeout parameter. In fact, performance with backup at the lower timeout setting clearly exceeds performance without backup at the higher time-out setting. The improvement comes at a cost of increase(\] cpu time, as can be seen in Figure 2; the increase is less for the higher value of the timeout parameter, even though the benefit to accuracy remains high.</Paragraph>
      <Paragraph position="2">  We investigated the effects of varying the timeout parameter when backup processing is in use. Recall that this parameter is the amount of cpu time allotted for eacl~ word of an utterance before timeout. Backup processing resets this allotment, adjusted for the current position in the utterance, so that the amount of time spent processing an utterance can increase by a factor of two to four over the initial allotment, depending on the number of keywords in the utterance.</Paragraph>
      <Paragraph position="3">  Figure 3 shows the results of our investigation. A setting of the timeout parameter below 0.3 is clearly undesirable. A setting of 0.3 enables the system to process correctly all but a handful of the utterances it could handle at a higher setting; that is, the curve changes at this point to a nearly horizontal orientation. At a setting of 1.1, the system achieves maximal performance accuracy. Somewhat surprisingly, if the utterances of each controller are considered separately, these findings remain the same, even though the content and phrasing of the utterances vary noticeably.</Paragraph>
      <Paragraph position="4"> The optimal setting of the timeout parameter depends on the relative costs of processing time and application errors. The 0.3 setting might be optimal for archival purposes or high volume processing. The 1.1 setting might be necessary for applications which demand maximal accuracy at any cost.</Paragraph>
    </Section>
    <Section position="2" start_page="33" end_page="34" type="sub_section">
      <SectionTitle>
5.2. Air Travel Information System
</SectionTitle>
      <Paragraph position="0"> As examples of data which our system handles properly, we list some inputs which are successfully processed. All of them were previously unseen test data from the November 1992 ATIS test. None of them would result in a parse from normal parsing. These inputs include false starts, corrections, constructions not covered by our grammar, and breaks in parsing due to unknown words.</Paragraph>
      <Paragraph position="1"> Our technique contributes for all these phenomena.</Paragraph>
      <Paragraph position="2"> I would like to do you have any flights between Philadelphia and Allanta (false start) Okay shoot I would have to choose the Delta flight nine seventy seven departing at twelve pm and arriving in San Francisco at two ten pm shoot and choose were unknown words, but the system recovers and understands the chosen flight.</Paragraph>
      <Paragraph position="3">  Okay American Airlines does it leave Philadelphia for Dallas in the mornings Left dislocation, not in our grammar; the airline is parsed as a fragment separate from the main body of the question, and semantic processing integrates the two parses correctly.</Paragraph>
      <Paragraph position="4"> Yes could you please give me a list of all American Airline first class flights lo from Philly to Dallas Fort Worth please The correction of the preposition at to from Philly is successfully handled by our technique; to is dropped, and parse fragments are produced for the rest of the input starting at from.</Paragraph>
      <Paragraph position="5"> Quantitative Results Because the semantic integration of fragmentary information is still in progress, the robust processing mechanism did not affect our final score on the ATIS evaluation. However, we did look closely at the effect of robust processing on parsing accuracy, in order to answer the following two questions: * How much does the backup mechanism improve parsing accuracy? * How often does the backup mechanism do the right thing? In order to answer the first question, we compared the proportion of usable or potentially usable non-X parses which the system produced with and without backup on the subset of the 1992 ATIS test collected at BBN.</Paragraph>
      <Paragraph position="6"> Without backup, 77% of the parses were usable; with backup, 88% were usable. Thus, backup resulted in an 11% increase in the number of usable parses.</Paragraph>
      <Paragraph position="7"> In order to answer the second question, we looked at the parses produced by backup. We found that 45% of them were usable or potentially usable by semantics. Of the parses that were not usable, we found that most of the time they were unusable because the system did not have information about some semantically important word in the sentence. Because of this missing information, the system ended up ignoring the word, and consequently the parse did not contain this important word. The fact that many of the unusable parses were due to lexical gaps was encouraging, because it means that the backup mechanism will continue to improve in this respect simply as new words are added to the system in the normal course of development.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>