<?xml version="1.0" standalone="yes"?>
<Paper uid="W03-0418">
  <Title>Identifying Events using Similarity and Context</Title>
  <Section position="3" start_page="0" end_page="0" type="metho">
    <SectionTitle>
2 An Event Correlation Learning System
</SectionTitle>
    <Paragraph position="0"> To understand the event formation process, some background on our larger system is useful. That system, SPANIEL (Sequence Processing and ANalysis for Inference Enhancement and Learning), attempts to learn ordered correlations between events from a corpus of parsed narratives.1 We assume that the narratives all describe related, but not necessarily identical, situations. An event is a particular atomic occurrence described by a single clause. SPANIEL uses a statistical model to identify correlations. Our overall goal is to capture significant correlations between events. We provide an overview here, and will discuss the details in a future paper.</Paragraph>
    <Paragraph position="1"> Briefly, SPANIEL uses a modified Markov model to capture correlations. The model attempts to capture a sequence identifying the significant events in a text. Our modification to Markov chaining captures the fact that events can be conditioned on a prior event that occurs several clauses distant, instead of on direct sequential dependence between events.</Paragraph>
    <Paragraph position="2"> After the training data is parsed, SPANIEL creates a graph from individual clauses. This graph is meant to capture the semantically important information from each clause in the training texts. The nodes in the graph are created from the actual clauses, with the arcs in the graph indicating which clauses occur together in a text, and in what order. Events are built from the nodes in the graph, with the arcs providing the starting point for correlations between events.</Paragraph>
    <Paragraph position="3"> We convert the parsed text to graph nodes using a frame representation. Each frame contains four slots.</Paragraph>
    <Paragraph position="4"> Three primary slots represent the agent, the action, and the theme of the clause. The agent and action slot fillers are straightforward: the agent slot takes a noun phrase that was tagged as the clause's subject by the parser, and the action slot takes the clause's verb phrase. The theme slot is more flexible, and can be filled by a noun phrase, prepositional phrase, or adjective phrase that follows the verb. Noun phrases tagged as the direct object have priority, but the first prepositional or adjective phrase can serve as the filler in the absence of a direct object.2 A fourth slot is only filled if the system identifies a dependent clause through the presence of a clause marker or the absence of a theme (e.g. &amp;quot;She reported she was having control problems.&amp;quot;). Words are not stemmed, except 1Following previous authors, we use narrative instead of story to refer to a text, indicating that we do not expect to find a full plot structure.</Paragraph>
    <Paragraph position="5"> 2Passive verbs result in rearrangement of the fillers.</Paragraph>
    <Paragraph position="6"> in two special cases described later.</Paragraph>
    <Paragraph position="7"> The three primary slots are filled by a head and a modifier. The head is the head word of the appropriate phrase. Prepositional phrases also concatenate the preposition to the head. Verb phrases include all verbs from the phrase, along with particles. The modifier is a concatenation of all adjectives, adverbs, and other nouns. Two examples of sentences and the resulting frames are given in Figure 1.</Paragraph>
    <Paragraph position="8"> We define three functions over the graph. These functions allow us to both detect correlations between events, and identify when nodes might represent the same event.</Paragraph>
    <Paragraph position="9"> First, the cooccurrence of two nodes, C(Ni; Nj), is the number of narratives in which Ni occurs before Nj, and thus quantifies each arc. Ni and Nj do not have to be neighbors: any number of other clauses could have separated them. Cooccurrence is directional: C(Ni; Nj) is not the same as C(Nj; Ni). Next, two nodes Ni and Nj match, denoted M(Ni; Nj), if their frames are identical.3 Finally, two nodes Ni and Nj are similar, denoted S(Ni; Nj), if they demonstrate a certain degree of textual similarity. This function plays an important role in event creation, and we define it in detail in the next section.</Paragraph>
    <Paragraph position="10"> Given this graph, SPANIEL uses a beam search to detect correlations of multiple events by incrementally expanding a sequence of events that occur frequently together. As of this writing, most resulting correlations include only a pair of events, with rare sequences of three events. Our goal is to find such correlations with minimal domain-specific knowledge. Thus, SPANIEL must create events from individual nodes as it expands event sequences. We describe this event formation process, the heart of this paper, in the next section.</Paragraph>
  </Section>
  <Section position="4" start_page="0" end_page="0" type="metho">
    <SectionTitle>
3 Event Construction and Revision
</SectionTitle>
    <Paragraph position="0"> The goal of event formation is to identify nodes that can play the same role within a domain. Event formation starts with a single node, identified as a potential part of a correlation, and attempts to expand this simple event by adding additional nodes. This process is unsupervised, so we restrict the search by defining two criteria that nodes must meet to be included in the same event. First, the Similarity criterion states that each node in an event must be textually similar to at least one other node in the event.</Paragraph>
    <Paragraph position="1"> Therefore, a node must be similar to at least one node that is already part of an event in order to be added to the event. Second, the InSeq criterion states that each node in the event must occur, in the appropriate order, with at least one node in each of the event's immediate neighbors in the sequence.</Paragraph>
    <Paragraph position="2"> Similarity is determined by a function S(Ni; Nj) that indicates whether two nodes share sufficient textual fea3Identical frames do not guarantee identical text, as the frames are simplified versions of the original clauses.</Paragraph>
    <Paragraph position="3"> The pilot reported that he was running out of fuel.</Paragraph>
    <Paragraph position="4">  tures. Two nodes satisfy this function if two of their three primary slots share heads, with some exceptions described below. Empty fillers are not considered similar. In addition, the nodes must have similar dependent clauses (identified by recursively applying the same criteria). Null fillers for this slot are considered similar. An example of two textually similar nodes is given in Figure 2.</Paragraph>
    <Paragraph position="5"> Two exceptions to this general definition handle nominalizations and &amp;quot;to be.&amp;quot; First, if two nodes' actions do not match, but the stemmed head of one node's action matches the stemmed head of the other node's theme, the nodes are considered similar. Second, if the stemmed form of a node's action is &amp;quot;to be&amp;quot;, the action is ignored in determining similarity. Therefore, the node's agent and theme must match the other node for the nodes to be similar. null The second criteria for adding nodes to events, In-Seq, ensures that all nodes in an event occur in similar contexts. Within a correlation, we call events immediately prior to or following a given event its neighbors. Neighbors provide contextual information indicating which nodes occur in similar places in narratives. The InSeq constraint states that for a node to be added to an event, it must cooccur at least once with each of that event's neighbors in the proper order. To test for this cooccurrence, we extend the definition of cooccurrence to apply to mixed node and event arguments. Thus C(N; E) or C(E; N) is the number of times node N cooccurred in the training data with any node from event E. The order in which the arguments are given is the order in which</Paragraph>
    <Paragraph position="7"> they must appear to count.</Paragraph>
    <Paragraph position="8"> An example of two nodes (N2 and N3) that meet our two criteria is given in Figure 3. The two nodes are both textually similar (S(N2; N3) = true) and have the same context (C(E1; N2), C(E1; N3), C(N2; E3), and C(N3; E3) are all greater than zero), so they can be combined into one event E2.</Paragraph>
    <Paragraph position="9"> Event formation is divided into two algorithms. The first, AUGMENT, adds nodes to an event based on the context provided by one neighbor. The second, REVISE, removes nodes that do not meet the InSeq criterion when an event acquires a second neighbor through further correlation expansion.</Paragraph>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
3.1 The AUGMENT Algorithm
</SectionTitle>
      <Paragraph position="0"> The job of the AUGMENT algorithm is to form a new event and add it to the front or back of a correlation.</Paragraph>
      <Paragraph position="1"> Therefore the new event initially has one neighboring event. Each new event is initialized from a single node, and AUGMENT adds all nodes that meet both the Similarity and InSeq criteria to this node.</Paragraph>
      <Paragraph position="2"> A phenomenon we refer to as self-cooccurrence complicates the augmentation process. This is when two nodes in the same event both occur in the same narrative.</Paragraph>
      <Paragraph position="3"> Since assigning two nodes to an event proposes that the nodes represent the same occurrence, self-cooccurrence is undesirable: it represents examples where the two Input: seed, a seed node, B, a neighbor event Output: the augmented event, E</Paragraph>
      <Paragraph position="5"> a different type of occurrence, we bias the cooccurrence function to penalize events that contain multiple nodes from the same narrative.</Paragraph>
      <Paragraph position="6"> SPANIEL employs the AUGMENT algorithm, illustrated in Figure 4, each time it evaluates the addition of a new event to a sequence. The new event is seeded with an initial node, and AUGMENT then checks each node that is similar to the event. If the node occurs at least once with the neighboring event and does not occur more frequently with the event being augmented, the algorithm adds the node to the event. We show the pseudocode for augmentation at the beginning of a sequence; augmentation at the conclusion of a sequence is analogous, with C(N; B) replaced by C(B; N) in all cases.</Paragraph>
      <Paragraph position="7"> Two special cases bootstrap event formation when a correlation of two events is first created. First, AUGMENT allows the InSeq criterion to be fulfilled by cooccurrence with a node similar to the existing neighbor, since that neighbor initially consists of a single node. Second, since that neighbor also previously had no context to constrain it, AUGMENT is also called on the neighbor.</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
3.2 The REVISE Algorithm
</SectionTitle>
      <Paragraph position="0"> As previously mentioned, AUGMENT only has one neighbor against which to enforce the InSeq criterion. Fulfillment of the criterion using one neighbor does not ensure that the node will be acceptable given a second neighbor.</Paragraph>
      <Paragraph position="1"> Therefore, if a second neighbor to this event is added to the sequence, the REVISE algorithm removes nodes that fail to remain acceptable. Figure 5 illustrates the situation with a sequence containing three events, the new one having just been added at the front of the sequence. Each node in the new event meets the InSeq criterion, but now a node in its neighbor does not. REVISE removes such nodes from events. Any removals from the first event revised may also warrant removals from the next neighbor in the sequence and so on. Therefore, SPANIEL applies REVISE to each existing node in the sequence in order,</Paragraph>
      <Paragraph position="3"> Input: An event sequence, S =&lt; E1; E2; : : : ; En &gt; Output: An event sequence respecting InSeq</Paragraph>
      <Paragraph position="5"> starting with the event neighboring the new event.</Paragraph>
      <Paragraph position="6"> REVISE (see Figure 6 for the pseudocode) checks that all nodes in an event are still valid members, given the context provided by a new neighbor. If the algorithm removes any nodes from the event, it proceeds to check the next event in order. The pseudocode presents this ordering from left to right, as when an event is added to the front end of the correlation. The reverse traversal is analogous, with the loop proceeding from n 1 down to 1 instead of from 2 up to n, where n is the length of the sequence. Note that the first (respectively last) event does not need to be checked, as AUGMENT ensures the InSeq criterion for this event with its single neighbor.</Paragraph>
      <Paragraph position="7"> REVISE can have one additional effect. If revision results in any event having no nodes, the proposed event addition fails. In other words, the new event is inconsistent with the rest of the correlation.</Paragraph>
      <Paragraph position="8"> In summary, the AUGMENT algorithm creates an event that contains all nodes that are similar to the original node and that cooccur with the neighboring event in the sequence. Since AUGMENT works on an event that has only one neighbor, REVISE checks these constraints when a second neighbor is added next to an event. Together, the algorithms enforce the two criteria we use to define events.</Paragraph>
    </Section>
  </Section>
  <Section position="5" start_page="0" end_page="0" type="metho">
    <SectionTitle>
4 Evaluation
</SectionTitle>
    <Paragraph position="0"> Automatic evaluation of individual events would be difficult, since they primarily have meaning in the context of a specific correlation. Therefore, we primarily employ manual evaluation. In addition, we describe a series of ablation tests that illustrate why events and our two event formation criteria are needed.</Paragraph>
    <Paragraph position="1"> To conduct the evaluation, we applied SPANIEL to a corpus of narratives from the National Transportation Safety Board's online database of summaries of aviation accidents. This database, as of our experiments, contains more than 43,000 texts. We sampled five sets of approximately 4,300 texts each, and subdivided these sets with eighty percent of the sample used for training and twenty percent used for testing.</Paragraph>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
4.1 Event Cohesion
</SectionTitle>
      <Paragraph position="0"> To evaluate event generation, we presented human judges with events acquired by our system and asked them to score each event by assessing the nodes' concurrence in describing an identifiable type of occurrence. The judges were asked to apply the following procedure:  1. Read all nodes in the event, and determine what occurrence the nodes describe.</Paragraph>
      <Paragraph position="1"> 2. Read all the nodes again, with the dominant concept of the occurrence in mind. For each node, assign a score of one if the node matches the concept or zero otherwise.</Paragraph>
      <Paragraph position="2"> 3. For each event, total the scores of the nodes. This  gives a score for the event.</Paragraph>
      <Paragraph position="3"> The first step is asks the judge to choose the dominant concept of each event based on their interpretation of the nodes. The second and third steps assign a score to each event based on the nodes it contains. We then divide this score by the number of nodes in the event to give a percentage of nodes that match the dominant concept of the occurrence represented by the event. We call these the conforming nodes.</Paragraph>
      <Paragraph position="4"> We used two human judges, both of whom have backgrounds in computer science and aviation. Each judge was given the same set of results from the learning system, with each two event correlation separated into individual events. An example of an event, printed as the  judges received it, is given in Figure 7. We generated correlations from the five different data sets, and blindly chose one set to give the judges. Figure 10 gives the distribution of the number of nodes in each event. To provide a baseline, ten percent of the events presented to the judges were generated by probabilistically choosing a set of random nodes from the graph, with more frequent nodes chosen with higher probability. Each random event contains between two and ten nodes. The judges were not informed of this procedure, and thus we can obtain a measure of their scoring applied to a randomly generated event.</Paragraph>
      <Paragraph position="5"> In Table 1, we give the mean fraction of conforming nodes for our event generation algorithm and the random baseline.5 We present more detailed results in two graphs, one for the results from each judge, in Figure 8 and Figure 9. We plot both the baseline and the learned results, giving the fraction of nodes that were judged conforming versus the size of the event in nodes. Where more than one event had the same number of nodes, we plot the mean conforming node fraction for the events. These graphs give an indication of the cohesiveness of the events as the size of the events increases. The cohesiveness of the learned events does not appear to drop dramatically as the size of the events increases, whereas the the baseline's performance does appear to decrease.</Paragraph>
      <Paragraph position="6"> 5Our second judge, instead of giving a numerical score and description of the event's dominant concept, simply put a question mark for each of the two baseline events of size 10. We interpret this result as no dominant concept, which equates to the existence of a pathological dominant concept of size one, and therefore one conforming node.</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
4.2 Ablation Testing
</SectionTitle>
      <Paragraph position="0"> We also evaluated the utility of events by eliminating SPANIEL's ability to acquire events. The ablated version of SPANIEL can only learn correlations of nodes, not events. We used the same five sets of training texts as input to the ablated system. The full system learned between 22 and 28 correlations meeting a probability threshold corresponding to approximately 12 occurrences in the training set, while the ablated system learned between zero and two correlations with the same probability. null We also ablated SPANIEL by removing each of the two criteria, Similarity and InSeq, individually. Removing either criteria rendered the system computationally infeasible to run. Neither version could expand even a single sequence in a reasonable amount of time. This phenomenon appears to be the result of two similar causes.</Paragraph>
      <Paragraph position="1"> If the InSeq criterion is disabled, the system encounters a chain reaction caused by similar nodes. Each time a node is added to an event, more nodes become similar to the event, causing uncontrolled expansion. If the Similarity criterion is disabled, any node that occurs in the proper order with the neighboring event can be added to the event. Since cooccurrence ignores intervening clauses, a potentially huge number of nodes can cooccur with an event, again causing uncontrolled expansion. These re- null sults support our hypothesis that, since event formation is an unsupervised process, it must be somewhat conservative to avoid allowing excessive noise into events.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>