<?xml version="1.0" standalone="yes"?>
<Paper uid="H92-1061">
  <Title>Fragment Processing in the DELPHI System</Title>
  <Section position="6" start_page="309" end_page="309" type="evalu">
    <SectionTitle>
5. RESULTS AND DISCUSSION
</SectionTitle>
    <Paragraph position="0"> This system was mn with the DELPI-I/ NL system in the February 1992 official evaluation. Using the same constant executable Lisp image (&amp;quot;disksave&amp;quot;) run for the official resuits, the test was run using a number of different switch settings, and scored with the version of the NIST comparator used for the official results. The switch conditions were: no fallback processing at all (which is simply the core DELPHI system), Syntactic Combiner only, Frame Combiner only, and both Syntactic Combiner and Frame Combiner working together (which was the condition used in the official results). The figures for NL only are reported in Table 1.</Paragraph>
    <Paragraph position="1">  Note the flame-only condition is actually better than result officially reported, in which both fallback sub-components were used.</Paragraph>
    <Paragraph position="2"> For the SLS test, the output ofBBN's BYBLOS N-best recognizer was used, with N = 5. The core DELPHI system (without fragments) was first tested against the five theories. If an intepretation was found for one of them, it was returned. Otherwise, the fallback methods were applied.</Paragraph>
    <Paragraph position="3"> Results for three of the four conditions are seen in Table 2 (results for the no-fallback = core DELPHI condition were unavailable as of this writing). The figure for the combination of both fragment modules (the configuration used in the official test) reflects an slight downward adjustment from the original value of 43.7 that corrects a purely procedural error committed during our running the test (the file that specifies &amp;quot;todays's date&amp;quot; for each query was not loaded, leading to a small increase in the number of wrong answers). This problem was fixed in obtaining the results in Table 2. As in the regular NL test, the SLS results show an noticeable improvement over the official results when the Frame Combiner is used alone.</Paragraph>
    <Paragraph position="4"> These results tend to undercut a central premise of our original strategy: namely that using both fragment combination methods together would improve the result over the use of either alone. Our tentative hypothesis is that the Syntactic Combiner, when failing and passing to the Frame Combiner the best results of its combination attempt, is passing wrongly combined fragments which mislead the Frame Combiner.</Paragraph>
    <Paragraph position="5"> On the other hand, these results do show the utility of the Frame Combiner when used alone. For NL only, it reduced the No Answer rate by 11.2 percentage points, and Weighted Error by 4.2 percentage points. For SLS, it reduced the Weighted Error from the adjusted official value of 43.2% to the signifigantly lower value of 39.2%.</Paragraph>
  </Section>
class="xml-element"></Paper>