<?xml version="1.0" standalone="yes"?>
<Paper uid="M91-1005">
  <Title>Category Total Labor Hours Domain Independent MUC-3 Specific Profiler 0 180 Parser / Analyzer Grammar rule s Lexical entrie s</Title>
  <Section position="2" start_page="49" end_page="49" type="intro">
    <SectionTitle>
MUC-3 RESULTS AND ANALYSIS
</SectionTitle>
    <Paragraph position="0"> Table 1 shows the official Total Slot Scores for TST2-MUC3 .</Paragraph>
    <Paragraph position="2"> Because of unanticipated system engineering issues associated with the scale-up to a lexicon of the size required for MUC-3, we were unable to run the parser/analyzer on the TST2 message set . The results therefore reflect just the output from the profiler. Since this was set up to provide input to the parser (rather than to perform the best template fill possible), our templates have only the incident-type slot filled. We could have used profiler output to fil l out some of the other slots to some degree of confidence less than could be obtained through a finer-grained analysis , but we did not do this because our strategy was to have the parser fill out these slots . This strategy might change in th e future, depending on the robustness of the parser in analyzing MUC-3 texts .</Paragraph>
    <Paragraph position="3"> The high overgeneration of templates based on profiler output was expected, given the processing strateg y explained above. Most of this overgeneration will be eliminated with the addition of the parser, assuming it can analyz e the sentences given to it as input by the controller .</Paragraph>
    <Paragraph position="4"> Our analysis of failures to correctly identify templates is summarized in Table 2 . The numbers in this table do not always sum to the total for a row because a single failure may have multiple causes .</Paragraph>
    <Paragraph position="5"> Failures appear to reduce to three types. First, we could have gained a few more points in both precision an d recall of the incident-type slot by correcting some template mappings . Although we did not realize this when we were scoring our results, the overgeneration resulted in some faulty mappings of optional templates to incorrect overgenerated templates.</Paragraph>
    <Paragraph position="6">  Second, the addition of the parser should, in theory, fix most of the incorrect and partially correct responses . The parser should, for example, fix some of the partially correct responses by resolving attacks into bombings or arso n by recognizing the incident-instrument relation, resolving attacks into murders by recognizing that death was cause d by the event, and resolving murders into death threats by recognizing the speech act-event relation. Most of the over-generated templates that resulted in incorrect mappings would also have been eliminated by recognizing that the description is too vague or the event does not meet one of the relevance criteria . We note, however, that the parser will not eliminate all of the overgeneration until we have developed appropriate modules for reference resolution and discourse structure analysis .</Paragraph>
    <Paragraph position="7"> A third type of failure, inadequacies of the profiler knowledge base, contributed primarily to missing templates. The types of profiler knowledge base faults are summarized in Table 3 .</Paragraph>
    <Paragraph position="8">  parse sentences with un recognized words lower detection threshold lower detection threshold add threat to profil e lower detection threshol d create concept tree from</Paragraph>
  </Section>
class="xml-element"></Paper>