<?xml version="1.0" standalone="yes"?>
<Paper uid="M92-1012">
  <Title>MCDONNELL DOUGLAS ELECTRONIC SYSTEMS COMPAN Y MUC-4 TEST RESULTS AND ANALYSIS</Title>
  <Section position="3" start_page="0" end_page="0" type="metho">
    <SectionTitle>
LIMITING FACTORS
</SectionTitle>
    <Paragraph position="0"> The greatest limiting factor for us again was time. Development of our new system TexUS reflects about one man-year of development and less than 4 man-months for the MUC4 task. Nevertheless, in a period of less than two months with a total of 3 persons, TexUS received a new and very thorough lexical processor and new baseline semantic and discourse processors . According to our development plans, th e performance ofthe system should match that of the best participants at MUC4 by the end of 1992 .</Paragraph>
  </Section>
  <Section position="4" start_page="0" end_page="0" type="metho">
    <SectionTitle>
TRAINING THE SYSTE M
</SectionTitle>
    <Paragraph position="0"> Extensive testing facilities were set up to automatically run and score TexUS on all 1300 message s of the development corpus. Testing was distributed on 5 to 9 Sun workstations (running in the background) processing 1300 messages within 6 to 10 hours. In addition, any 100 message set could be distributed, processed, and scored within one hour.</Paragraph>
    <Paragraph position="1"> Another program for testing key phrases, linguistic phenomena, and slot fills, was also set up t o monitor the consistency and progress of TexUS . These tests consist of pairs of small hand-made MUC4 templates and their corresponding key-templates. The current version of TexUS is then run on these hand-made templates and the differences printed out to monitor progress. One advantage of this testing format is that performance on individual slots is easy to isolate .</Paragraph>
    <Paragraph position="2"> Due to a lack of time and incompleteness of the current system, the testing facilities were no t extensively used prior to the official testing . The few times they were used on all 1300 messages of the development set, the results provided useful feedback. These tools, in conjunction with the development set, will be used extensively between MUC4 and MUC5 to monitor and speed development of our system .</Paragraph>
  </Section>
  <Section position="5" start_page="0" end_page="0" type="metho">
    <SectionTitle>
SUCCESSES AND NON-SUCCESSE S
</SectionTitle>
    <Paragraph position="0"> Our greatest success was in the fact that our system is now an end-to-end system . Last year wit h only the pattern matching algorithm complete, we quickly reached the limitations of such a capability an d our score could not go much higher without MUC-specific prodding. This year's system, TexUS, being a complete end-to-end system, has great potential to grow and score much higher than last year's system.</Paragraph>
    <Paragraph position="1"> The lack of development time was the major cause of our difficulties . We needed more time to add knowledge and expand the linguistic coverage of the system.</Paragraph>
  </Section>
  <Section position="6" start_page="0" end_page="114" type="metho">
    <SectionTitle>
REUSABILIT Y
</SectionTitle>
    <Paragraph position="0"> One of TexUS' greatest strengths is its portability to new domains. TexUS not only is written in C and can be compiled into any C or C++ program, but it also is domain-independent. Only three customization tasks were needed to perform the MUC4 task : identification of key words and concepts, special semantic classification of words for the MUC4 domain, and a post-processing module to convert generic output to MUC4 output.</Paragraph>
    <Paragraph position="1"> Less than 10 percent of the current code is dedicated to MUC4-like processing, 90 percent of whic h consists of template generation and merging as dictated by MUC4 template guidelines. The time needed to customize TexUS to a new domain is a matter of a few man-months, as demonstrated by our system during MUC3 and MUC4. Development time is cut further by the use of our graphic interface tools .</Paragraph>
  </Section>
  <Section position="7" start_page="114" end_page="115" type="metho">
    <SectionTitle>
LESSONS LEARNED
</SectionTitle>
    <Paragraph position="0"/>
  </Section>
class="xml-element"></Paper>