<?xml version="1.0" standalone="yes"?>
<Paper uid="M92-1006">
  <Title>GE ADJUNCT TEST REPORT : OBJECT-ORIENTED DESIGN AND SCORING FOR MUC- 4</Title>
  <Section position="4" start_page="0" end_page="79" type="metho">
    <SectionTitle>
MOTIVATIO N
</SectionTitle>
    <Paragraph position="0"> There are a variety of reasons why an object-oriented design is desirable . First, an object-oriented design is conceptually easier to understand . Instead of a flat listing of all slots of a template, slots pertaining to a single fill are grouped together. Cross-references are no longer needed, as indentation indicates the grouping, so the visual design is cleaner.</Paragraph>
    <Paragraph position="1"> Asthetics aside, there are additional performance data that can be obtained when groups of slots ar e connected as objects . Systems can be scored on how well a given object aligns, in a way analogous t o template matching alignment scores (template ID score) . Moreover, it is possible to construct an object total that does not consider whether any given object appears in a matching template or not . For exampl e in our system, we found that data such as a human target was correctly extracted by the program with all it s associated fields, but was put in the wrong template . This resulted in missing and spurious points for all th e  Story slot: incident</Paragraph>
    <Section position="1" start_page="78" end_page="79" type="sub_section">
      <SectionTitle>
Incident
</SectionTitle>
      <Paragraph position="0"> slot: date slot: location slot: type slot: stage slot: inst id slot: inst type slot: perp id slot: perp org slot: INSTRUMENT slot: perp conf slot: category slot: phys tgt id slot: PERPETRATOR slot: TARGET slot:hum tot num slot: phys tot num slot: phys tgt type slot: phys tgt number slot: phys tgt for nat slot: phys tgt effect slot: hum tgt name slot: hum tgt desc slot: hum tgt type slot: hum number slot: hum for nat slot: hum tgt effect Figure 1 : Object-oriented MUC-4 Template Desig n fields in the object, although it could be argued that all that was incorrect was the association of the object with an incorrect incident . Finally, with objected-oriented totals, it is very easy to isolate performanc e problems down to the object-level . With a flat design, less-than-perfect templates must be examined t o determine where the problems occurred . With object matching totals, it is possible to immediately isolat e object-level errors which facilitates the error diagnosis process . The rest of this paper maps out the processes used to transform the flat MUC-4 template design to a n object-oriented design . We then overview the scoring experiments we performed to test the effect of variou s configurations . Finally, we present detailed analyses of the effect of object-oriented design and scoring o n the data from MUC-4 systems performance .</Paragraph>
    </Section>
  </Section>
  <Section position="5" start_page="79" end_page="79" type="metho">
    <SectionTitle>
TRANSFORMATION TO 0-0 DESIGN
</SectionTitle>
    <Paragraph position="0"> The first step in performing this test was to automatically transform the existing template design to the object-oriented design illustrated in Figure 1. This process was aided by the existing cross-references, but was complicated by a variety of special cases we encountered. These special cases fell into two general categories; system glitches, and problems with the MUC-4 template format. The first three problems described below are system glitches; the last three are issues in the design of the templates.</Paragraph>
    <Paragraph position="1">  1. Inconsistent cross-references: We encountered inconsistent cross-reference strings. For example, ' 'PEOPLE' J may have been present as a human target description, but a slot intended to cross-reference to it may have read &amp;quot;TWO PEOPLE j.</Paragraph>
    <Paragraph position="2"> 2. Violation of Template Filling Rules: Some sites cross-referenced the perpetrator confidence to a null value, or to the PERP ID slot for example.</Paragraph>
    <Paragraph position="3"> 3. Multiple Set Fills: We encountered fills such as NO INJURY NO DEATH INJURY DEATH.</Paragraph>
    <Paragraph position="4"> 4. Inconsistent treatments: The treatment of repeated fills, as could be required in sentences such as &amp;quot;KILLED 3 PEOPLE AND INJURED 2 a was handled in different ways, with PEOPLE repeated as a fill, or with two EFFECTS cross-referenced to one fill.</Paragraph>
    <Paragraph position="5"> . When an EFFECT has a blank value, its scoping is ambiguous. When we attempt 5. Ambiguity of ' ' - ' '. to group targets into objects, we cannot decide which object this effect belongs to.</Paragraph>
    <Paragraph position="6"> 6. Ambiguity of Optional Fills: It is impossible with the current template design to determine when an optional fill of an optional object was meant, as opposed to a required fill of an optional object.  For the system problems, we manually intervened and allowed the conversion process to proceed. However one sites' responses were too unusual too allow us to transform the output without a great deal of manual interaction. For the template design problems, we came up with adequate methods of working around the problems.</Paragraph>
  </Section>
  <Section position="6" start_page="79" end_page="79" type="metho">
    <SectionTitle>
OBJECT-ORIENTED SCORING
</SectionTitle>
    <Paragraph position="0"> After all (execept one) of the sites' answers were transformed into the object-oriented design, a modified version of the MUC-4 scoring program was run in a variety of configurations to test the effect of enforcing object-level matching. This program used the merged history file, and took 10 seconds to score an average run. It took one person-week to convert all the sites' answer templates to the object oriented design, and create a new version of the scoring program to use this design.</Paragraph>
    <Paragraph position="1"> We experimented with a variety of conditions for aligning templates and objects. These were:  1. Only incident type match. This was closer to the MUC-2 scoring conditions.</Paragraph>
    <Paragraph position="2"> 2. Must match on incident type, plus either a match on ID or type for target or ID or ORG for perpetrator. This duplicated the MUC-4 scoring, in that either a match on target or perpetrator would cause the incident to align.</Paragraph>
    <Paragraph position="3"> 3. Must match on incident type plus a match on the string ID slot of a target. With this condition, we only aligned templates if the targets aligned according to a stricter matching condition. This matching condition required at least a partial match on the target string. Note that virtually no templates have no targets.</Paragraph>
    <Paragraph position="4"> 4. Free-floating object match. For this design, we computed the score if objects were allowed to match each other without considering if they happen to belong to an aligning template object. That is, if a  system mistyped an ATTACK as an ARSON but correctly extracted any human or physical targets, credit would be given for these objects.</Paragraph>
  </Section>
  <Section position="7" start_page="79" end_page="79" type="metho">
    <SectionTitle>
DISCUSSIO N
</SectionTitle>
    <Paragraph position="0"> One of the advantages of object-oriented scoring is that it is possible to obtain object alignment totals, an d object matching totals . Figure 6 illustrates this type of data for our system on MUC-3 (a description of ou r system and a summary of our performance can be found in this volume) . The META SLOT table contains a measurement of how well our system aligned objects .</Paragraph>
    <Paragraph position="1"> The OBJECT-TOT table is useful to compare the totals for object matches when objects that appear in different templates are not scored, with the FF-OBJECT TOT table which presents the &amp;quot;free-floating&amp;quot; objec t totals, allowing for matches between unaligned objects that appear in incorrect templates to contribute t o recall and precision . The PSEUDO TOT table gives the pseudo-object numbers presented at the bottom of our score report for comparison .</Paragraph>
  </Section>
class="xml-element"></Paper>