<?xml version="1.0" standalone="yes"?>
<Paper uid="H92-1094">
  <Title>AUGMENTING WITH SLOT FILLER RELEVANCY SIGNATURES DATA</Title>
  <Section position="3" start_page="0" end_page="457" type="metho">
    <SectionTitle>
AUGMENTED RELEVANCY
SIGNATURES
</SectionTitle>
    <Paragraph position="0"> One shortcoming of relevancy signatures is that they do not take advantage of the slot fillers in the concept nodes.</Paragraph>
    <Paragraph position="1"> For example, consider two similar sentences: (a) &amp;quot;a civilian was killed by guerrillas&amp;quot; and (b) &amp;quot;a soldier was killed by guerrillas&amp;quot;. Both sentences are represented by the same relevancy signature: (killed, $murder-pass-1) even though sentence (a) describes a terrorist event and sentence (b) does 1 According to the MUC-3 domain guidelines, events that targetted military personnel or installations were not considered to be terrorist in nature.</Paragraph>
    <Paragraph position="2"> not. To address this problem, we experimented with augmented relevancy signatures that combine the original relevancy signatures with slot filler information.</Paragraph>
    <Paragraph position="3"> Given a set of training texts, we parse each text and save the concept nodes that are generated. For each slot in each concept node 2, we collect reliability statistics for triples consisting of the concept node type, the slot name, and the semantic feature of the filler. 3 For example, consider the sentence: &amp;quot;The mayor was murdered.&amp;quot; The word &amp;quot;murdered&amp;quot; triggers a murder concept node that contains &amp;quot;the mayor&amp;quot; in its victim slot. This concept node instantiation yields the slot triple: (murder, victim, ws-govemment-official). For each slot triple, we then update two statistics: \[1\] the number of times that it occurred in the training set (N), and \[2\] the number of times that it occurred in a relevant text (NR). The ratio of NR over N gives us a &amp;quot;reliability&amp;quot; measure. For example, .75 means that 75% of the instances of the triple appeared in relevant texts.</Paragraph>
    <Paragraph position="4"> Using these statistics, we then extract a set of &amp;quot;reliable&amp;quot; slot triples by choosing two values: a reliability threshold Rslot and a minimum number of occurrences threshold Mslot. These parameters are analogous to the relevancy signature thresholds. The triples that satisfy the reliability criteria become our set of &amp;quot;reliable&amp;quot; slot filler triples. The algorithm for classifying texts is fairly simple. Given a new text, we parse the text and save the concept nodes that are produced during the parse, along with the words that triggered them. For each concept node, we generate a (triggering word, concept node) pair and a set of slot triples. If the (triggering word, concept node) pair is in our list of relevancy signatures, and the concept node contains a reliable slot triple then we classify the text as relevant. If not, then the text is deemed irrelevant. Intuitively, a text is classified as relevant only if it contains a strong relevancy cue and the concept node enabled by this cue contains at 2We only collect statistics for top-down slots, i.e. slots that were predicted by the concept node.</Paragraph>
    <Paragraph position="5"> 3Since slot fillers can have multiple semantic features, we create one triple for each feature. For example, if a murder concept node contains a victim with semantic features ws-human &amp; ws-military then we create two triples: (murder, victim, ws-human) and (murder, victim, ws-military).</Paragraph>
  </Section>
  <Section position="4" start_page="457" end_page="457" type="metho">
    <SectionTitle>
COMPARATIVE EXPERIMENTS
</SectionTitle>
    <Paragraph position="0"> We compared the performance of the augmented relevancy signatures with the original Relevancy Signatures Algorithm in order to measure the impact of the slot filler data. We tested the augmented relevancy signatures on the same two test sets that we had isolated for our original experiments, after training on the remaining 1300 texts.</Paragraph>
    <Paragraph position="1"> Figure 1 shows the original results produced by the Relevancy Signatures Algorithm and Figure 2 shows the results produced by the augmented relevancy signatures.</Paragraph>
    <Paragraph position="2"> Each data point represents a different combination of paramemr values.</Paragraph>
    <Paragraph position="3"> These graphs clearly show that the augmented relevancy signatures perform at least as well as the original relevancy signatures on these two test sets. The. most striking difference is the improved precision obtained for DEV 801900. There are two important things to notice about Figure 2. First, we are able to obtain extremely high precision at low recall values, e.g., 8% recall with 100% precision and 23% recall with 90% precision. Relevancy signatures alone do not achieve precision greater than 67% for this test set at any recall level. Second, although there is a very scauered distribution of data points at the lower recall end, we see consistently better precision coupled with the higher recall values. This trend suggests that the augmented relevancy signatures perform at least as well as the original relevancy signatures when they are working with statistically significant numbers of texts.</Paragraph>
    <Paragraph position="5"> Furthermore, the Relevancy Signatures Algorithm demonstrated extremely strong performance on DEV ~01500 and it is reassuring to see that the augmented relevancy signatures achieve similar results, perhaps even showing a slight improvement at the higher recall values. The highest recall level obtained with extremely high precision by the original relevancy signatures was 67% with 98% precision.</Paragraph>
    <Paragraph position="6"> The augmented relevancy signatures achieved significantl.v higher recall with nearly the same precision, 77% recall with 96% precision.</Paragraph>
  </Section>
class="xml-element"></Paper>