<?xml version="1.0" standalone="yes"?>
<Paper uid="M92-1011">
  <Title>Precision A Matched/ Missing</Title>
  <Section position="3" start_page="108" end_page="423" type="metho">
    <SectionTitle>
RESULTS
</SectionTitle>
    <Paragraph position="0"> The complete LSI TST3 and TST4 score reports are included in Appendix G, &amp;quot;Final Test Score Summaries&amp;quot; .</Paragraph>
    <Paragraph position="1"> As an indication of system development during MUC4, we can compare our TST3 results with our results on th e MUC-4 interim test (TST2) . The relevant portions of the TST3 and TST2 results are shown in Tables 2 and 3 .</Paragraph>
    <Paragraph position="2"> Figure 1 graphically presents the TST2 and TST3 recall and precision matrices .</Paragraph>
    <Paragraph position="3">  772 5  Although our overall TST3 and TST4 scores clearly fell short of our goals, there are important comparisons t o be made between TST2 and TST3 . Most importantly, our recall scores made a definite improvement, as can be seen in the TST3 REC column (vs. the TST2 REC column) and on the Recall axis in Figure 1 . This is due to improvements in the extraction of events and entities, from the text, at our knowledge representation level . (Some examples of this are given in the system summary paper in our discussion of Message 0048) . Unfortunately, our precision did not significantly improve . This is in large part due to template overgeneration, which is caused by deficiencies in our event template merging . We are not yet properly merging event references across multiple sentences .</Paragraph>
    <Paragraph position="4"> Although improvements in both recall and precision are required, we anticipate that first solving the overgeneration problem will give us a more accurate picture of how the system is really performing in terms of recall an d  iio precision, and where additional work will produce the most significant improvement in system performance .</Paragraph>
  </Section>
  <Section position="4" start_page="423" end_page="423" type="metho">
    <SectionTitle>
ALLOCATION OF EFFOR T
</SectionTitle>
    <Paragraph position="0"> Figure 1 of LSI's system summary in this proceedings presents an overview of the DBG system as configure d for MUC-4. A new module has been added at the front end to select sentences of potential interest for th e application. Work on our Principle-based parser has continued throughout the past year, extending the inventor y of syntactic structures that can currently be handled .</Paragraph>
    <Paragraph position="1"> The major MUC-4 effort was devoted to the lexicon (approximately 35%) and to the parser (about 20%), wit h other modules getting substantially less of the total effort, as shown in Table 4 .</Paragraph>
  </Section>
  <Section position="5" start_page="423" end_page="423" type="metho">
    <SectionTitle>
LIMITING FACTORS
</SectionTitle>
    <Paragraph position="0"> MUC is unfortunately a resource-limited undertaking for LSI; however, we did expend a significant effort on the lexicon and parser for MUC-4. Although LSI is a small company, we were able to devote these resources t o MUC-4 in part due to the sponsorship of DARPA and BRL (see Footnote 1), and additionally, because the work was directly in line with our overall NLP objectives mentioned previously.</Paragraph>
    <Paragraph position="1"> Limiting factors included all those on the list -- time, people, cpu cycles -- as well as the budgetary limits mentioned above. Knowledge was also a limiting factor in the sense that portions of the knowledge embedded i n the system were not exploited, and other crucial knowledge was not added, due to resource limitations.</Paragraph>
    <Paragraph position="2"> On the other hand, the amount of knowledge represented in the expanded lexicon is significant, so significan t achievements are possible if limited resources are focused on particular problem areas .</Paragraph>
  </Section>
  <Section position="6" start_page="423" end_page="423" type="metho">
    <SectionTitle>
TRAINING
</SectionTitle>
    <Paragraph position="0"> During our preparation for MUC-4 testing, we were able to use the entire development corpus this year, and found it extremely valuable in our system development .</Paragraph>
  </Section>
  <Section position="7" start_page="423" end_page="423" type="metho">
    <SectionTitle>
MODULE MOST OVERDUE FOR REWRITIN G
</SectionTitle>
    <Paragraph position="0"> The code for our Lexical Unexpected Inputs/Word Acquisition Module (LUX/WAM), which deals with erroneous (e.g., misspelled) or new words is still the one which has gone for the longest period of time without rewriting or optimization of any kind. However, with our new, much larger lexicon, LUX/WAM was invoked far les s frequently than during MUC-3 processing, and so was not really a significant factor in MUC-4 .</Paragraph>
    <Paragraph position="1"> A second module mentioned last year as a candidate for rewriting was LXI, the lexical lookup component .</Paragraph>
    <Paragraph position="2"> Some modification of LXI code was carried out to provide more efficient processing for MUC-4 .</Paragraph>
  </Section>
class="xml-element"></Paper>