<?xml version="1.0" standalone="yes"?>
<Paper uid="M92-1038">
  <Title>University of Massachusetts : Description of the CIRCUS System a s Used for MUC-4</Title>
  <Section position="4" start_page="284" end_page="284" type="metho">
    <SectionTitle>
AT A LIGHT ON A STREET IN DOWNTOWN SAN SALVADOR &gt;CO ANINDIVIDUALPLACEDABOMB ON THE ROOF
OF THE ARMORED VFRTCLE &gt;PE)
</SectionTitle>
    <Paragraph position="0"> CIRCUS generates two CNs here. One fairly complicated CN is triggered by &amp;quot;PLACED .&amp;quot; This CN picks up not just the bomb as a weapon, but also the individual as the responsible party, and the vehicle as a target . The second CN describes the bomb as a weapon and its link to the targeted vehicle (as before) . These two CNs are largely redundant, and they are merged into a single incident structure because they share the same partition . This incident structure contains a perpetrator id = &amp;quot;AN INDIVIDUAL&amp;quot; along with the following subeven t  should be merged, thereby picking up a physical target for the first time . Had we picked up this physical target from S3 as well, the target integration test would have merged the two vehicle descriptions at this point as well . Note that MBC merges the description of the perpetrator as &amp;quot;an individual&amp;quot; with the previously encountered descriptor &amp;quot;urban guerrillas&amp;quot; because the earlier description is recognized to be more specific .</Paragraph>
    <Paragraph position="1"> S9-10: (we omit these sentences from the discussion - no alterations to memory are made )</Paragraph>
  </Section>
  <Section position="5" start_page="284" end_page="285" type="metho">
    <SectionTitle>
Si!:(GUERRTLLAS ATTACKF,D (dMF.RTNOPS HOMR TN SAN SALVADOR ON APR 14 89 &gt;CO &amp;&amp;5 DAYS AGO &gt;CO
WITH EXPLOSIVES &gt;PE)
</SectionTitle>
    <Paragraph position="0"> CIRCUS generates 7 highly redundant CNs in response to S11 . The most comprehensive CN instantiates a n attack with actor = &amp;quot;GUERRILLAS,&amp;quot; target = &amp;quot;MERINO'S HOME,&amp;quot; and instrument = &amp;quot;EXPLOSIVES.&amp;quot; This same  CN also picks up the location (San Salvador) and date (April 14) by the bottom-up attachment mechanism . Locations and dates are normally not predicted by CN definitions, but they can be inserted into available CNs vi a bottom-up attachment. All of this information is incorporated into a single incident structure containing a bombin g subevent (an attack using explosives is understood to be a bombing) . The resulting incident structure is then passed to the memory integration portion of MBC .</Paragraph>
    <Paragraph position="1"> Just as before, MBC checks to see if the new incident can be merged into the lone incident structure currentlystored in memory . But this time the new structure fails to match the existing structure because of incompatibl e targets. MBC cannot merge a home with a vehicle. When MBC fails to merge the new bombing incident with th e old bombing incident, it moves down the target stack to see if there is another incident structure that might merge , but there are no other physical targets in memory . MBC adds the new incident to the top of the incident stack, and memory now contains two bombing incidents .</Paragraph>
    <Paragraph position="2"> S12:(THERE WERE &amp;&amp;7 CHILDREN &gt;CO INCLUDING &amp;&amp;4 OF THE VICE @PRESIDENT@S CHILDREN &gt;CO IN THE</Paragraph>
  </Section>
  <Section position="6" start_page="285" end_page="285" type="metho">
    <SectionTitle>
HOME AT THE TIME &gt;PE)
</SectionTitle>
    <Paragraph position="0"> CIRCUS produces no output for this sentence because no CN triggers are encountered . We sometimes miss information in sentences where the only verb is a form of &amp;quot;to be.&amp;quot; S13:(A 75--YEAR--OLD NIECE OF @MERINO@S WAS INJURED &gt;PE) CIRCUS generates an injury CN with victim = &amp;quot;A 15-YEAR-OLD NIECE .&amp;quot; This results in a subevent o f unknown type with a victim id = &amp;quot;A 15-YEAR-OLD NIECE.&amp;quot; When MBC receives this incident, it examines the first incident on the top of its stack to see if a merge is possible. Since no incompatible victims are found i n memory for this incident (the latest bombing incident specifies no victims), a merging occurs .</Paragraph>
    <Paragraph position="1"> S14-S17 : [we omit these sentences from our discussion - no alterations are made to memory .] S18: (RICARDO VALDIVIESO &gt;CO PRESIDENT OF THE LEGISLATIVE ASSEMBLY AND AN ARENA LEADER &gt;CO</Paragraph>
  </Section>
  <Section position="7" start_page="285" end_page="286" type="metho">
    <SectionTitle>
SAID THE FMLN AND ITS FRONT GROUPS ARE RESPONSIBLE FOR THE &amp;quot;IRRATIONAL VIOLENCE THAT JTLLED
ATTORNRY GRNF.RAT, GARCIA &gt;DQ &gt;PE)
</SectionTitle>
    <Paragraph position="0"> CIRCUS produces a murder CN with victim = &amp;quot;Attorney General Garcia&amp;quot; and actor = &amp;quot;irrational violence .&amp;quot; This CN has a soft constraint on the actor slot which specifies a human or organization, but the CN survives the CN filter because its other variable slot has a filler that does meet the required soft constraints (the filter errs on the side of spurious information if one slot looks good and the other slot looks bad) . MBC is careful to check available soft constraints when it integrates information into its preliminary incident structures . Any slot fill that violates a soft constraint is discarded at that time .</Paragraph>
    <Paragraph position="1"> When MBC attempts to integrate this incident into memory, it locates a compatible target in the victi m stack, and merges the new incident structure with the existing structure that describes Garcia as a victim . Because we have now merged new information into an incident that was not at the top of the incident stack, we have to reorde r the incident stack by moving the most recently referenced incident to the top of the stack . This effectively identifies the first incident as the current topic once again . Ideally, this would set us up to correctly integrate information contained later in S21 and S22 where new information is presented about the vehicle bombing, but CIRCUS fails to pick up the additional human targets from those sentences, so the topic shift that we 've successfully recognized at S18 goes unrewarded .</Paragraph>
    <Paragraph position="2"> When MBC completes its analysis, the two bombing incident structures are converted into two template instantiations, along with a third threat incident picked up from additional sentences near the end of the text . In order to instantiate the final templates, we rely on semantic features in our dictionary to recognize a home as a civilia n residence and an armored vehicle as a transport vehicle. When instantiating response templates, we attempt to fill all slots with the exception of phys-tgt-total-num and hum-tgt-total-num.</Paragraph>
    <Paragraph position="3"> We did fairly well on the first template (see Figure 1) . We missed San Salvador as the location within E l Salvador, we said the vehicle was destroyed instead of damaged, and we missed 3 human targets (the driver who wa s not hurt, and the 2 bodyguards, one of whom was injured). All the other slots were correctly filled . On the second template, we fail in three places. We have no perpetrator organization, we miss the physical target type for Merino' s  home (it should have been GOVERNMENT OFFICE OR RESIDENCE), and we are missing the 7 children tha t were human targets (this is one of the few texts where a hum-tgt-total-num slot should receive a value) .</Paragraph>
    <Paragraph position="4"> Overall, TST2-MUC4-0048 showed the UMass/MUC-4 system working fairly well and not making an y major errors. Most of our recall loss resulted from a failure to recognize relevant information in S12 (the 7 children) , S21 and S22 (the driver and 2 bodyguards). As we saw in this message, we can recover from some failures i n sentence analysis when a text provides redundant descriptions (e .g. we missed the physical target in S3, but picked it up correctly in S8). When memory-based consolidation responds correctly to topic transitions, the output tha t CIRCUS generates usually makes it into the correct places in the response templates . TST2-MUC4-0048 shows how MBC was able to correctly recognize two topic transitions: first from an old incident to a new incident, and then back again to the earlier incident . Given that the errors encountered for TST2-MUC4-0048 were relatively minor (one could even argue that the third template was valid and should have been covered by an optional key template), there is nothing here that illustrates the more damaging problems that impacted our TST3 and TST4 score reports .</Paragraph>
    <Paragraph position="5"> Figure 2 shows score reports for the two templates that mapped to TST2-MUC4-0048 answer keys, along wit h the score report for the entire message which averages in the spurious template that we generated for the threat .</Paragraph>
    <Paragraph position="6"> This final score report for the whole message illustrates how much negative impact spurious templates have o n precision if a system is generating one spurious template for every two good templates . If we had generated a summary score report based on only two templates instead of three, our All Templates precision would have been 94. With the third template averaged in, our All Templates precision drops to 76 .</Paragraph>
    <Paragraph position="7">  In a domain that is characterized by complicated domain guidelines, and lots of grey areas, answer keys cannot be trusted to give encodings that are necessarily superior to the output of a high performance extraction system. If this is the case, it may be very difficult to attain 85% precision under all templates, and optimal precision level s may be closer to the 70-80% range.</Paragraph>
  </Section>
class="xml-element"></Paper>