<?xml version="1.0" standalone="yes"?>
<Paper uid="M91-1022">
  <Title>Assuming C-BOMBING-TEMPLATE and C-INJURY-TEMPLATE have same R-INSTRUMENT Assuming C-BOMBING-TEMPLATE and C-DAMAGE-TEMPLATE have same R-INSTRUMENT Assuming C-BOMBING-TEMPLATE and C-INJURY-TEMPLATE have same R-DATE Assuming C-BOMBING-TEMPLATE and C-DAMAGE-TEMPLATE have same R-DATE Assuming C-BOMBING-TEMPLATE and C-INJURY-TEMPLATE have same R-PERPETRATOR Assuming C-BOMBING-TEMPLATE and C-DAMAGE-TEMPLATE have same R-PERPETRATOR Assuming C-BOMBING-TEMPLATE and C-INJURY-TEMPLATE have same R-TARGET Assuming C-BOMBING-TEMPLATE and C-DAMAGE-TEMPLATE have same R-TARGET Comparison of Program Answers with Answer Key The NLTooLsET results for TST1-0099 were the following templates :</Title>
  <Section position="1" start_page="0" end_page="0" type="metho">
    <SectionTitle>
GE:DESCRIPTION OF THE
NLTOOLSET SYSTEM AS USED FO RMUC- 3
</SectionTitle>
    <Paragraph position="0"/>
  </Section>
  <Section position="2" start_page="0" end_page="0" type="metho">
    <SectionTitle>
GE Research and Developmen t
</SectionTitle>
    <Paragraph position="0"/>
  </Section>
  <Section position="3" start_page="0" end_page="0" type="metho">
    <SectionTitle>
Abstract
</SectionTitle>
    <Paragraph position="0"> The GE NLTooLSET is a set of text interpretation tools designed to be easily adapted to new domains. This report summarizes the system and its performance on the MUG-3 task.</Paragraph>
  </Section>
  <Section position="4" start_page="0" end_page="144" type="metho">
    <SectionTitle>
INTRODUCTIO N
</SectionTitle>
    <Paragraph position="0"> The GE NLTooLsET aims at extracting and deriving useful information from text using a knowledge-based , domain-independent core of text processing tools, and customizing the existing programs to each new task .</Paragraph>
    <Paragraph position="1"> The program achieves this transportability by using a core knowledge base and lexicon that adapts easil y to new applications, along with a flexible text processing strategy that is tolerant of gaps in the program' s knowledge base.</Paragraph>
    <Paragraph position="2"> The NLTooLSET's design provides each system component with access to a rich hand-coded knowledg e base, but each component applies the knowledge selectively, avoiding the computation that a complete analysis of each text would require . The architecture of the system allows for levels of language analysis , from rough skimming to in-depth conceptual interpretation .</Paragraph>
    <Paragraph position="3"> The NLTooLSET, in its first version, was behind GE 's participation in the MUCK-II conference . Since MUCK-II, the Toolset, now in Release 2 .1, has expanded to include a number of new capabilities, includin g a text pre-processor for easier customization and better performance, broader lexical and syntactic coverage , and a domain-independent module for applying word-sense preferences in text . In addition to being teste d in several new application areas, the Toolset has achieved about a 10 times speedup in words per minute s over MUCK-II, and can now partially interpret and tag word senses in arbitrary news stories, although it i s very difficult to evaluate this task-independent performance . These basic enhancements preceded the other additions, including a discourse processing module ;, which were made for MUC-3 .</Paragraph>
    <Paragraph position="4"> The performance of the program on tasks such as MUCK-II and MUC-3 derives mainly from two design characteristics: central knowledge hierarchies and flexible control strategies . A custom-built 10,000 word-root lexicon and 1000-concept hierarchy provides a rich source of lexical information . Entries are separated b y their senses, and contain special context clues to help in the sense-disambiguation process . A morphologica l analyzer contains semantics for about. 75 affixes, and can automatically derive the meanings of inflecte d entries not separately represented in the lexicon . Domain-specific words and phrases are added to th e lexicon by connecting them to higher-level concepts and categories present in the system 's core lexicon and concept hierarchy. Lexical analysis can also be restricted or biased according to the features of a domain .</Paragraph>
    <Paragraph position="5"> This is one aspect of the NLTooLSET that makes it highly portable from one domain to another .</Paragraph>
    <Paragraph position="6"> The language analysis strategy in the NLTooLSET uses fairly detailed, chart-style syntactic parsin g guided by conceptual expectations . Domain-driven conceptual structures provide feedback in parsing, contribute to scoring alternative interpretations, help recovery from failed parses, and tie together information across sentence boundaries . The interaction between linguistic and conceptual knowledge sources at the leve l of linguistic relations, called &amp;quot;relation-driven control&amp;quot; was a key system enhancement before MUC-3 .</Paragraph>
    <Paragraph position="7"> In addition to flexible control, the design of the NLTooLSET allows each knowledge source to influenc e different stages of processing . For example, discourse processing starts before parsing, although many decisions about template merging and splitting are made after parsing . This allows context to guide languag e analysis, while language analysis still determines context.</Paragraph>
    <Paragraph position="8">  The next section briefly describes the major portions of the NLTooLSET and its control flow ; the remainder of the paper will discuss the application of the Toolset to the MUC-3 task .</Paragraph>
  </Section>
  <Section position="5" start_page="144" end_page="147" type="metho">
    <SectionTitle>
SYSTEM OVERVIEW
</SectionTitle>
    <Paragraph position="0"> Processing in the NLTooLSET divides roughly into three stages : (1) pre-processing, consisting mainly o f a pattern matcher and discourse processing module, (2) linguistic analysis, including parsing and semantic interpretation, and (3) post-processing, or template filling . Each stage of analysis applies a combination o f linguistic, conceptual, and domain knowledge, as shown in Figure 1 .</Paragraph>
    <Section position="1" start_page="144" end_page="147" type="sub_section">
      <SectionTitle>
Pre-processing Analysis Post-processing
</SectionTitle>
      <Paragraph position="0"> The pre-processor uses lexico-semantic patterns to perform some initial segmentation of the text, identifying phrases that are template activators, filtering out irrelevant text, combining and collapsing som e linguistic constructs, and marking portions of text that could describe discrete events . This component is described in [1] . Linguistic analysis combines parsing and word sense-based semantic interpretation with domain-driven conceptual processing . The programs for linguistic analysis are largely those explained i n [2, 3]--the changes made for MUC-3 involved mainly some additional mechanisms for recovering from failed processing and heavy pruning of spurious parses . Post-processing includes the final selection of template s and mapping semantic categories and roles onto those templates . This component used the basic element s from MUCK-II, adding a number of specialized rules for handling guerrilla warfare, types, and refines th e discourse structures to perform the template splitting and merging required for MUC-3 .</Paragraph>
      <Paragraph position="1"> The control flow of the system is primarily from linguistic analysis to conceptual interpretation to domai n interpretation, but there is substantial feedback from conceptual and domain interpretation to linguistic analysis. The MUC-3 version of the Toolset includes our first implementation of a strategy called relation-drive n control, which helps to mediate between the various knowledge sources involved in interpretation . Basically, relation-driven control gives each linguistic relation in the text (such as subject-verb, verb-complement, o r verb-adjunct) a preference score based on its interpretation in context . Because these relations can apply t o a great many different surface structures, relation-driven control provides a means of combining preference s without the tremendous combinatorics of scoring many complete parses . Effectively, relation-driven control permits a &amp;quot;beam&amp;quot; strategy for considering multiple interpretations without producing hundreds or thousands of new paths through the linguistic chart .</Paragraph>
      <Paragraph position="2"> The knowledge base of the system, consisting of a feature and function (unification-style) gramma r with associated linguistic relations, and the lexicon mentioned earlier, still proves transportable and largel y generic . The core lexicon contains over 10,000 entries, of which 37 had to be restricted because of specialize d usage in the MUC-3 domain (such as device, which always means a bomb, and plant, which as a verb usually means to place a bomb and as a noun usually means the target of an attack) . The core grammar contains about 170 rules, with 50 relations and 80 additional subcategories . There were 23 MUC-specific additions to this grammatical knowledge base, including 8 grammar rules, most of them dealing with unusual nou n phrases that describe organizations in the corpus .</Paragraph>
      <Paragraph position="3"> 14 5 The control, pre-processing, and transportable knowledge base were all extremely successful for MUG3; remarkably, lexical and grammatical coverage, along with the associated problems in controlling searc h and selecting among interpretations, proved not to be the major stumbling blocks for our system--furthe r distinguishing events and merging or splitting templates proved to be the major obstacle in obtaining a better score .</Paragraph>
      <Paragraph position="4"> ANALYSIS OF TST1-009 9 The common &amp;quot;walkthrough&amp;quot; example, TST1-0099, is a good example of many of the problems in analysi s and template filling, although it is somewhat unrepresentative of the difficulties in parsing because the ke y content is contained in fairly simple sentences. We will explain briefly what our program did, then provide details of the story-level and sentence-level interpretation with an analysis of the templates produced .</Paragraph>
      <Paragraph position="5"> Overview of Exampl e In many ways, TST1-0099 is representative of how the Toolset performed on MUC-3 . The program parsed most of the key sentences, failed to parse some of the less relevant sentences, missed a key relationshi p between locations--thus failing to split a template into two separate events--and incorrectly included a n earlier bombing as part of a main event in the story. One additional program fill was scored incorrect becaus e the answer key had the wrong date . The program thus derived 36 slots out of a possible 43, with 21 correct , 2 partial, 2 incorrect, and 11 spurious, for 51% recall, 61% precision, and 30% overgeneration .</Paragraph>
      <Paragraph position="6"> Detail of Message Ru n As explained in the previous section, the Toolset uses pattern matching for pre-processing, followed b y discourse processing, parsing and semantic interpretation, and finally template-filling .</Paragraph>
      <Paragraph position="7"> Pre-processing uses pattern matching to manipulate the input text . It recognizes relevant fillers, tags , collapses constructs, and segments the text into fragments describing different events . For example, th e pattern matcher recognizes the phrase describing the bombing event in the first sentence of the text, collapses the conjunctive phrase the embassies of the PRC and the Soviet Union, and marks that as a complementize r (rather than a relative pronoun, pronoun, or determiner) . In later sentences, it also marks locative phrases like in the Lima residential district of San Isidro and located in Orrantia district . The discourse processin g module does an initial text segmentation based on (1) definite and indefinite references like a car bomb and the attack, (2) the relationship between events (e .g. bombing and arson), and (3) cue phrases . This identifies  six events : (1) the phrase bombed describes a bombing event ; the phrase the bombs marks its continuation ; (2) the phrase a car bomb exploded signifies a new bombing event ; (3) the temporal cue phrase meanwhile combined with the phrase two bombs in indefinite form signifie s another bombing event ; the phrases the bombs and the attacks mark the continuation of this event ; (4) the phrase a Shining Path bombing indicates yet another bombing event ; the sentence gets delete d because the temporal information in the phrase some three years ago violates MUC-3 constraints; (5) the cue phrase in another incident combined with the phrases killed and dynamite delineates anothe r bombing event; the sentence gets deleted because the temporal information from the phrase three years ag o violates MUC-3 constraints; (6) the phrase burned indicates a new arson event .</Paragraph>
      <Paragraph position="8">  Linguistic analysis parses each sentence and produces (possibly alternative) semantic interpretations at the sentence level . These interpretations select word senses and roles, heavily favoring domain-specific senses . Post-processing maps the semantic interpretations onto templates, eliminating invalid fills (in this cas e none), combining certain multiple references (such as to the embassies), and adding certain information (lik e adding numbers and &amp;quot;guessing&amp;quot; terrorist groups as fillers when it fails to find evidence to the contrary) . Post-processing collapses three of the segments (events) produced during pre-processing into one template .  Sentence-level Interpretatio n The following is the trace of the first two sentences of TST1-0099 : The &amp;quot;call&amp;quot; to Trumpet represents the en d of the first stage of semantic interpretation and the beginning of conceptual analysis, and the output tha t follows this represents the mapping (or role extension) from semantic roles to templates .</Paragraph>
      <Paragraph position="10"/>
    </Section>
  </Section>
  <Section position="6" start_page="147" end_page="147" type="metho">
    <SectionTitle>
2. INCIDT-DATE 24 OCT 89
3. INCIDT-TYPE ARSON
4. CATEGORY TERRORIST ACT
5. INDIV-PERPS
6. ORG-PERPS &amp;quot;SHINING PATH&amp;quot;
7. PERP-CONF REPORTED AS FACT: &amp;quot;SHINING PATH&amp;quot;
8. PHYS-TGT-ID &amp;quot;BUSES&amp;quot;
</SectionTitle>
    <Paragraph position="0"> The failure to recognize that San Isidro and Orrantia are distinct locations caused the program to combin e the two bombings into one template (under the mistaken assumption that Orrantia is in San Isidro) . We do not know now why the program did not fill in the City name &amp;quot;Lima&amp;quot;, although this would not have affecte d the score . As a result of the location assumption, the Toolset got two extra fills for slots 8 and 10 in th e first template (effectively by merging the templates), missed slot 9 entirely (because the number of target s is different), and got one extra fill in slot 14, in addition to partial credit for the location .</Paragraph>
    <Paragraph position="1"> The program correctly discarded the bombing of the bus in 1989 but failed to group the 15 wounde d Soviet marines correctly with that event (because of a simple bug which caused the deletion of the earlie r event before the wounding effect was processed), thus losing points also in Template 1, slots 11, 12, 13, 14 , and 18 (even though slot 18 was correct except for the cross-reference) .</Paragraph>
    <Paragraph position="2"> The program got the correct date from tonight, October 25 . The answer key has a range, October 24-25 .</Paragraph>
    <Paragraph position="3"> The second template produced by the Toolset was completely correct, but the score for this message is 51% precision and 61% recall, mainly due to the combining of two possible templates into one .</Paragraph>
  </Section>
class="xml-element"></Paper>