File Information
File: 05-lr/acl_arc_1_sum/cleansed_text/xml_by_section/metho/98/m98-1029_metho.xml
Size: 49,232 bytes
Last Modified: 2025-10-06 14:14:48
<?xml version="1.0" standalone="yes"?> <Paper uid="M98-1029"> <Title>Source: Lynette Hirschman (mailto:lynette@MITRE.org)</Title> <Section position="2" start_page="0" end_page="1" type="metho"> <SectionTitle> 2. GENERAL NOTATION </SectionTitle> <Paragraph position="0"/> <Section position="1" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 2.1 SGML Tagging </SectionTitle> <Paragraph position="0"> The annotation for coreference is SGML tagging within the text stream. Referring expressions and their antecedents are tagged as follows:</Paragraph> <Paragraph position="2"> The basic annotation contains the information to establish some type of link between an explicitly marked pair of noun phrases. In the above example, the pronoun &quot;it&quot; is tagged as referring to the same entity as the phrase, &quot;Lawson Mardon Group Ltd.&quot; There is one markup per string. Other links can be inferred from the explicit links. We assume that the coreference relation is symmetric and transitive, so if phrase A is marked as coreferential with B (indicated by a REF pointer from A to B), we can infer that B is coreferential with A; if A is coreferential with B, and B is coreferential with C, we can infer that A is coreferential with C.</Paragraph> </Section> <Section position="2" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 2.2 The &quot;TYPE&quot; Attribute </SectionTitle> <Paragraph position="0"> The purpose of the TYPE attribute is to indicate the relationship between the anaphor and the antecedent. At present only one such relationship, &quot;IDENT&quot; (for identity), is being annotated.</Paragraph> </Section> <Section position="3" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 2.3 The &quot;ID&quot; and &quot;REF&quot; Attributes </SectionTitle> <Paragraph position="0"> The ID and REF attributes are used to indicate that there is a coreference link between two strings. The ID is arbitrarily but uniquely assigned to the string during markup. The REF uses that ID to indicate the coreference link.</Paragraph> </Section> <Section position="4" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 2.4 The &quot;MIN&quot; Attribute </SectionTitle> <Paragraph position="0"> The MIN attribute is used in the answer key (&quot;key&quot;) to indicate the minimum string that the system under evaluation must include in the COREF tag in order to receive full credit for its output (&quot;response&quot;). So, in the next example, if the system response had omitted &quot;of Surrey, England&quot; from the COREF tag, the response would nonetheless receive full credit because it identified the minimum string.</Paragraph> <Paragraph position="1"> <COREF ID=&quot;100&quot; MIN=&quot;Haden MacLellan PLC&quot;>Haden MacLellan PLC of Surrey, England</COREF> ... <COREF ID=&quot;101&quot; TYPE=&quot;IDENT&quot; REF=&quot;100&quot;>Haden MacLellan</COREF> Any response which includes the MIN string and does not include any tokens beyond those enclosed in the <COREF>...</COREF> tags is valid. The MIN string will in general be the HEAD of the phrase; see section 5 for a full discussion of this issue. Note that only the annotation KEY distinguishes between the maximal string and the MIN string; the response key does not have a MIN attribute.</Paragraph> </Section> <Section position="5" start_page="0" end_page="1" type="sub_section"> <SectionTitle> 2.5 The &quot;STATUS&quot; Attribute </SectionTitle> <Paragraph position="0"> The STATUS (&quot;status&quot;) attribute is used in the answer key when the markup is optional. The only value for this attribute is OPT (&quot;optional&quot;).</Paragraph> <Paragraph position="1"> The evaluation software will not score a string that is marked OPT in the key unless the response has markup on that string. A potential example is given below. (It is marked OPT because a reader may not be certain that &quot;Livingston Street&quot; refers to the Board of Education.) Note that the optionality is marked only for the anaphor.</Paragraph> <Paragraph position="2"> At the Feb. 96 meeting of the Coreference and Ellipsis working group, the suggestion was made to distinguish markups that are optional because of textual ambiguity from markups that are optional because of unclear or missing markup guidelines. Although this seems a workable suggesion, a little experimentation may be advisable before implementation.</Paragraph> <Paragraph position="3"> Our <COREF ID=&quot;102&quot; MIN=&quot;Board of Education&quot;>Board of Education</COREF> budget is just too high, the Mayor said. <COREF ID=&quot;103&quot; STATUS=&quot;OPT&quot; TYPE=&quot;IDENT&quot; REF=&quot;102&quot;>Livingston Street</COREF> has lost control.</Paragraph> </Section> </Section> <Section position="3" start_page="1" end_page="1" type="metho"> <SectionTitle> 3. WHAT PART OF THE TEXT TO ANNOTATE </SectionTitle> <Paragraph position="0"> Coreference markup should be made on the body of the text and on corpus-specific portions of the header. The SGML tags that are used to identify the body and the various portions of the header may vary from one corpus to another.</Paragraph> <Section position="1" start_page="1" end_page="1" type="sub_section"> <SectionTitle> 3.1 Specific Guidance for MUC-7 Corpus </SectionTitle> <Paragraph position="0"> The annotation of coreference is to be performed within the text delimited by the SLUG, DATE, NWORDS, PREAMBLE, TEXT, and TRAILER tags.</Paragraph> </Section> <Section position="2" start_page="1" end_page="1" type="sub_section"> <SectionTitle> 3.2 Specific Guidance for Speech Transcriptions </SectionTitle> <Paragraph position="0"> If the transcript contains disfluencies or verbal erasures, the &quot;erased&quot; portion should not be annotated for coreference; this means that it will be helpful to have the input text annotated for disfluencies before beginning coreference annotation, so that there is agreement on what is &quot;verbally deleted&quot; and what is part of the final output.</Paragraph> </Section> </Section> <Section position="4" start_page="1" end_page="1" type="metho"> <SectionTitle> 4. WHAT THINGS TO ANNOTATE </SectionTitle> <Paragraph position="0"/> <Section position="1" start_page="1" end_page="1" type="sub_section"> <SectionTitle> 4.1 Markables </SectionTitle> <Paragraph position="0"> The coreference relation will be marked between elements of the following categories: NOUNS, NOUN PHRASES, and PRONOUNS. Elements of these categories are MARKABLES. PRONOUNS include both personal and demonstrative pronouns, and with respect to personal pronouns, all cases, including the possessive. Dates (&quot;January 23&quot;), currency expressions (&quot;$1.2 billion&quot;), and percentages (&quot;17%&quot;) are considered noun phrases.</Paragraph> <Paragraph position="1"> A noun phrase is markable whether it is the object of an assertion, a negation, or a question. Thus, &quot;a machete&quot; is markable in all of the following examples: I have a machete.</Paragraph> <Paragraph position="2"> I don't have a machete.</Paragraph> <Paragraph position="3"> Do you have a machete? Note in particular that the initial introduction of an object into the discourse may often occur as an indefinite noun phrase (&quot;Do you have a machete?&quot; or &quot;I saw *a truck*; *it* turned the corner...&quot;). Also note that just because an element is &quot;markable&quot;, it does not follow that there are later references to it -- that is, it may or may not participate in coreference. That may even be true for pronouns -- section 4.5 for further discussion. Interrogative &quot;wh-&quot; noun phrases are NOT markables, e.g. &quot;Which engine&quot; and &quot;Who&quot; in the following queries: Which engine would you like to use? Who is your boss? The relation is marked only between pairs of elements both of which are markables. This means that some markables that look anaphoric will not be coded, including pronouns, demonstratives, and definite NPs whose antecedent is a clause rather than a markable. For example, in Program trading is &quot;a racket,&quot; complains Edward Egnuss, a White Plains, N.Y., investor and electronics sales executive, &quot;and *it's not to the benefit of the small investor*, *that*'s for sure.&quot; Though &quot;that&quot; is related to &quot;it's not to the benefit of the small investor&quot;, the latter is not markable, so no antecedent is annotated for &quot;that&quot;.</Paragraph> </Section> <Section position="2" start_page="1" end_page="1" type="sub_section"> <SectionTitle> 4.2 Terminology for Mark-Up </SectionTitle> <Paragraph position="0"> It is useful to define some terms to support a discussion of the difficult coreference issues. This section defines the terms &quot;extensional descriptor&quot;, &quot;intensional descriptor&quot; and &quot;grounding instance&quot;.</Paragraph> <Paragraph position="1"> An extensional descriptor is an enumeration of the member(s) of a set by (unique) names. In the context of the coreference task definition, this amounts to the use of proper names, e.g., *Jane Z. Smith*, *Chrysler Corporation*, or numerical values (The stock price was *$4.02*).</Paragraph> <Paragraph position="2"> An intensional description is a predicate that is true of an entity or set of entities -- that it, it characterizes or defines the members of the set: &quot;the prime numbers&quot;, &quot;the president of Chrysler Corporation&quot;. Any nonconcrete common noun, taken on its own, is an intensional description: it functions at the &quot;type&quot; level (&quot;president&quot;, &quot;problem&quot;) or, if it takes a quantifiable value, at the &quot;function&quot; level (&quot;rate&quot;, &quot;temperature&quot;). Intensional descriptions are useful for sets which have no finite extension (&quot;the set of odd numbers&quot;), or in cases where we don't know the extension (&quot;the gene sequence responsible for encoding the immune response&quot;).</Paragraph> <Paragraph position="3"> They can also be used to refer to instances of the type (*my Ford*... *the car*), or to values of the function (the [per share value of [$4.02]] ... [The stock price]).</Paragraph> <Paragraph position="4"> The grounding instance in a coreference chain is the first extensional description in the chain (most often, the first element in the chain). This terminology is useful in the discussion about function-value relations, time-dependent entities and bare nominals. Thus in the sequence Henry Higgins, who was formerly sales director for Sudsy Soaps, became president of Dreamy Detergents we have a sequence consisting of the extensional description *Henry Higgins* (which is the grounding instance), together with two intensional descriptions, *sales director for Sudsy Soaps* and *president of Dreamy Detergents*. In addition, there are two other extensional descriptions, *Sudsy Soaps*, and *Dreamy Detergents*.</Paragraph> <Paragraph position="5"> In the sentence The temperature rose to 90 degrees before dropping to 70 degrees.</Paragraph> <Paragraph position="6"> we have a function, &quot;temperature&quot;, which takes on a value (&quot;90 degrees&quot;) at one point in time, and at a later point in time, a second value, &quot;70 degrees&quot;. Because there is only one occurrence of the noun phrase &quot;the temperature&quot;, we have a problem marking the coreferring expressions. &quot;The temperature&quot; is a function expression, and is grounded first by &quot;90 degrees&quot;; an implicit second occurrence is coreferential with &quot;70 degrees&quot;. However, if we mark all of these as coreferential, we find that &quot;90 degrees&quot; is IDENT with &quot;70 degrees&quot;, which is clearly wrong. What we want to say is: 90 degrees instantiates &quot;The temperature&quot; at time t1; 70 degrees instantiates &quot;The temperature&quot; at time t2; And these are both of type &quot;temperature&quot; (but not IDENT).</Paragraph> <Paragraph position="7"> In section 1.3, we proposed conventions that prevent the collapsing of coreference chains, at the expense of losing some type coreference. Given that our mark-up conventions are already incomplete (we don't mark verb coreference), this seems like a small price to pay for making the chains we do mark useful in other information extraction tasks, e.g., the Template Element task. We provide marked-up examples of this in section 6.4.</Paragraph> </Section> <Section position="3" start_page="1" end_page="1" type="sub_section"> <SectionTitle> 4.3 Names and Other Named Entities </SectionTitle> <Paragraph position="0"> Names and other Named Entities (as defined in the MUC-6 document titled &quot;Named Entity Task Definition&quot; -dates, times, currency amounts, and percentages) are all markables. A substring of a Named Entity, however, is not a markable. Thus in *London* ... *London*-based ...</Paragraph> <Paragraph position="1"> the two instances of London are to be marked coreferential; in *Reuters Holding PLC* ... *Reuters* announced that &quot;Reuters Holding PLC&quot; and &quot;Reuters&quot; are to be marked coreferential. But in Equitable of Iowa Cos. ... located in Iowa.</Paragraph> <Paragraph position="2"> the two instances of &quot;Iowa&quot; are NOT to be marked as coreferential since the first is not a markable: it is a substring of a Named Entity. In addition to names as defined for the Named Entity task, other identifiers that are, in the opinion of the annotator, clearly not decomposable should be treated as atomic as well, e.g., &quot;Widener Library&quot; and &quot;E two&quot; in <COREF ID=&quot;0&quot; MIN=&quot;building&quot;>the large strange-looking building, which is <COREF ID=&quot;1&quot; TYPE=&quot;IDENT&quot;>Widener Library</COREF></COREF> and in okay then I'll take <COREF ID=&quot;0&quot; MIN=&quot;E two&quot;>engine E two</COREF> ... so uh the plan is to take <COREF ID=&quot;1&quot; TYPE=&quot;IDENT&quot; REF=&quot;0&quot; MIN=&quot;E two&quot;>engine E two</COREF> ... Date expressions recognized by the Named Entity task are also treated as atomic; components of a date are not separate markables. Thus, in In a report issued January 5, 1995, the program manager said that there would be no new funds this year. no relation is to be marked between &quot;1995&quot; and &quot;this year&quot;.</Paragraph> </Section> <Section position="4" start_page="1" end_page="1" type="sub_section"> <SectionTitle> 4.4 Gerunds </SectionTitle> <Paragraph position="0"> Gerunds (verbal forms using a present participle) are not markable. In *Slowing the economy* is supported by some Fed officials; *it* is repudiated by others. one should not mark the relation between &quot;slowing the economy&quot; and &quot;it&quot;. A phrase headed by a present participle is taken to be verbal if it can take an object (as in the above example) or can be modified by an adverb.</Paragraph> <Paragraph position="1"> Present participles which are modified by other nouns or adjectives (&quot;program trading&quot;, &quot;excessive spending&quot;), are preceded by an article (&quot;a&quot;, &quot;the&quot;, &quot;my&quot;, etc.) or are followed by an &quot;of&quot; phrase (&quot;slowing of the economy&quot;) are to be considered noun-like and ARE markable.</Paragraph> </Section> <Section position="5" start_page="1" end_page="1" type="sub_section"> <SectionTitle> 4.5 Pronouns </SectionTitle> <Paragraph position="0"> The possessive forms of pronouns used as determiners are markable. Thus in its chairperson there are two potential markables for relations: &quot;its&quot; and the entire NP, &quot;its chairperson&quot;. Similarly, in &quot;the man's arm&quot;, there are two markables, &quot;[the] man&quot; and &quot;the man's arm&quot;. The general question of what is to be treated as a lexical token (apostrophes in this case) is discussed in the MUC document titled &quot;Tokenization Rules.&quot; First, second, and third-person pronouns are all markable, so in &quot;There is no business reason for *my* departure&quot;, *he* added.</Paragraph> <Paragraph position="1"> &quot;my&quot; and &quot;he&quot; should be marked as coreferential. Reflexive pronouns are markable, so in *He* shot *himself* with *his* revolver.</Paragraph> <Paragraph position="2"> &quot;He&quot;, &quot;himself&quot;, and &quot;his&quot; should all be marked coreferential. Emphatics are also markable; thus, &quot;himself&quot; should also be marked coreferential, so that &quot;He&quot; and &quot;himself&quot; are marked coreferential in: *He* is, *himself*, unsure of the outcome.</Paragraph> <Paragraph position="3"> In certain cases, pronouns may not have an antecedent (&quot;It's raining&quot;) or they may refer to something unmarkable, for example, a clausal construction -- see section 4.1 above.</Paragraph> </Section> <Section position="6" start_page="1" end_page="1" type="sub_section"> <SectionTitle> 4.6 Bare Nouns </SectionTitle> <Paragraph position="0"> Prenominal modifiers (e.g., *ocean drilling* in &quot;the ocean drilling company&quot;) are markable only if either the prenominal modifier is coreferential with a named entity or to the syntactic head of a maximal noun phrase.</Paragraph> <Paragraph position="1"> That is, there must be one element in the coreference chain that is a head or a name, not a modifier. Thus the following instance is markable, because the prenominal modifier &quot;aluminum&quot; is coreferential with the head noun &quot;aluminum&quot; in the phrase &quot;market for aluminum&quot;.</Paragraph> <Paragraph position="2"> The price of *aluminum* siding has steadily increased, as the market for *aluminum* reacts to the strike in Chile.</Paragraph> <Paragraph position="3"> Similarly, the following two occurrences of &quot;drug&quot; would be marked: He was accused of money laundering and *drug* trafficking. However, the trade in *drugs*....</Paragraph> <Paragraph position="4"> Contrast this with the following occurrences of &quot;contract&quot; and &quot;contract drilling&quot; which would NOT be marked, because there are no occurrences of this phrase, except as a prenominal modifier in the following sequence: Ocean Drilling & Exploration Co. will sell its *contract drilling* business. ... Ocean Drilling said it will offer 15% to 20% of the *contract drilling* business through an initial public offering in the near future.</Paragraph> <Paragraph position="5"> Note that the occurrences of *its*, *its contract drilling business* and *the contract drilling business* would all be markable -- see section 6.5.</Paragraph> <Paragraph position="6"> While nouns in prenominal positions are sometimes markable, the noun which appears at the head of a noun phrase is not separately markable -- it is markable only as part of the entire noun phrase. Thus in the passage Linguists are a strange bunch. Some linguists even like spinach.</Paragraph> <Paragraph position="7"> it would not be correct to link the two instances of &quot;linguists&quot;.Similarly, in the sentence: The rate, which was 6 percent, was higher than that offered by the other bank.</Paragraph> <Paragraph position="8"> the noun phrase &quot;the rate&quot; is a function expression, instantiated by the predicate &quot;6 percent&quot;, so these two would be marked coreferential, as follows: <COREF ID=&quot;0&quot; MIN=&quot;rate&quot;>The rate, which was <COREF ID=&quot;1&quot; REF=&quot;0&quot;>6 percent</COREF>,</COREF> was higher than that offered by the other bank.</Paragraph> <Paragraph position="9"> In this example, pronoun *that* is coreferential at the FUNCTION level with *The rate*. However, *that* occurs as the head of a noun phrase, *that offered by the other bank*, which is NOT coreferential with *The rate* and *6 percent* (indeed, it refers to a higher rate), so *that* is an instance of a pronoun that cannot be marked in our current framework, even though we lose some type coreference information by not marking it.</Paragraph> </Section> <Section position="7" start_page="1" end_page="1" type="sub_section"> <SectionTitle> 4.7 Implicit Pronouns </SectionTitle> <Paragraph position="0"> Assume that English has no zero pronouns; in other words, the empty string is not markable. In Bill called John and spoke with him for an hour.</Paragraph> <Paragraph position="1"> there is no relation between the implicit subject of &quot;spoke&quot; and &quot;Bill&quot;. Do not code relations between a relative pronoun and the NP to which it attaches or the gap that it fills. Thus, in the movie which I saw the relative pronoun &quot;which&quot; bears no markable relation to either &quot;the movie&quot; (the head to which the relative pronoun attaches) or to the implicit object of &quot;saw&quot; (the gap that the pronoun fills).</Paragraph> </Section> <Section position="8" start_page="1" end_page="1" type="sub_section"> <SectionTitle> 4.8 Conjoined Noun Phrases </SectionTitle> <Paragraph position="0"> Noun phrases which contain two or more heads (as defined in section 4.1) are marked by defining the MINimal string (see section 5) as the span from the first &quot;head&quot; through the last &quot;head&quot; including all material in between. The MAXimal string includes the entire maximal conjoined noun phrase. Thus we mark coreference between &quot;The sleepy boys and girls&quot; and &quot;their&quot; as follows: <COREF ID=&quot;1&quot; MIN=&quot;boys and girls&quot;>The sleepy boys and girls</COREF> enjoy <COREF ID=&quot;2&quot; REF=&quot;1&quot; TYPE=&quot;IDENT&quot;>their</COREF> breakfast.</Paragraph> <Paragraph position="1"> In addition, the individual conjuncts are markable if they are separately coreferential with other phrases:</Paragraph> </Section> </Section> <Section position="5" start_page="1" end_page="1" type="metho"> <SectionTitle> 5. HOW MUCH OF THE MARKABLE TO ANNOTATE </SectionTitle> <Paragraph position="0"> The task is defined in order to allow maximal latitude for systems in identifying markables, and to decouple the evaluation from that of accurately parsing noun phrases. Accordingly, the string generated by a system to identify a markable must include the head of the markable (as defined below) and may include any additional text up to a maximal noun phrase (as defined below).</Paragraph> <Paragraph position="1"> In preparing the key, the text element to be enclosed in SGML tags is the maximal noun phrase; the head will be designated by the MIN attribute.</Paragraph> <Paragraph position="2"> [We expect that in the future it may be possible, when separate noun phrase bracketings are available, to automatically generate the maximal NP markup from a markup using only heads.]</Paragraph> <Section position="1" start_page="1" end_page="1" type="sub_section"> <SectionTitle> 5.1 Head of a Phrase </SectionTitle> <Paragraph position="0"> For most noun phrases, the head will be the main noun, without its left and right modifiers.</Paragraph> <Paragraph position="1"> If the head is a name, the entire name is marked. This includes suffixes such as &quot;Sr.&quot;, &quot;III&quot;, etc. on personal names and &quot;Corp.&quot; on organization names; it does not include personal titles or any modifiers. We follow in this regard the rules for marking personal and organization names for the Named Entity task, as well as for other non-geographic names (e.g, &quot;New Year&quot;): For constructions that are idioms or collocations, the minimal phrase will ignore the fact that this is a collocation and use the syntactic head; this is because the definition of a collocation is often domain-specific. In the following examples, the MIN is indicated by asterisks: income *taxes* light *year* *run* of the mill If the maximal noun phrase is the same as the head, the MIN need not be marked. Also, if the maximal noun phrase differs from the MIN only by the articles &quot;a&quot; or &quot;the&quot;, the MIN need not be marked, because the scoring program will automatically strip these before comparing answers.</Paragraph> </Section> <Section position="2" start_page="1" end_page="1" type="sub_section"> <SectionTitle> 5.2 Maximal Noun Phrase </SectionTitle> <Paragraph position="0"> The maximal noun phrase includes all text which may be considered a modifier of the noun phrase. This includes (among other modifiers) appositional phrases, non-restrictive relative clauses, and prepositional phrases which may be viewed as modifiers of the noun phrase or of a containing clause: *Mr. Holland* *the senior of the executives who will assume Holland's duties* *the rumor that the war had ended* *Fred Frosty, the ice cream king of Tyson's Corner,* *the Penn Central Co., which used to run a railroad,* XYZ Inc. formed *a joint venture with Sony* Note that in the fourth and fifth cases the final comma may be viewed as part of the NP, and so is included in the maximal NP. The system does not need to worry about punctuation, since the scorer strips punctuation before comparing key to response. In the last case, &quot;with Sony&quot; could equally well be taken to modify &quot;venture&quot; or &quot;formed&quot;, and so is included as part of the maximal NP around &quot;venture&quot;. Note also that in the &quot;Fred Frosty&quot; example, there is a coreference between the entire noun phrase and the appositional phrase, &quot;the ice cream king of Tyson's Corner&quot;; see section 6.3 for a discussion of this construct. In the case of a conjoined noun phrase with shared complements or modifiers, the maximal noun phrase for the conjoined phrase is the maximal noun phrase. The minimal noun phrase will begin the at the minimal phrase for the first conjunct and include everything up to the end of the minimal phrase for the last conjunct.</Paragraph> <Paragraph position="1"> If the conjuncts are referenced individually, the maximal noun phrases will NOT include the conjunct. The maximal NP for the first conjunct will include all of the NP up to the conjunction; the maximal NP for the second conjunct will include all of the NP following the conjunction: <COREF ID=&quot;1&quot; MIN=&quot;Fribble&quot;>Ms. Fribble</COREF> was <COREF ID=&quot;2&quot; REF=&quot;1&quot; TYPE=&quot;IDENT&quot; STATUS=&quot;OPT&quot;>president</COREF> and <COREF ID=&quot;3&quot; REF=&quot;1&quot; TYPE=&quot;IDENT&quot; STATUS=&quot;OPT&quot; MIN=&quot;CEO&quot;> CEO of Amalgamated Text Processing Inc.</COREF> It is possible that the maximal span of the noun phase is interrupted by material that is not part of the noun phrase. Such discontinuous noun phrases should nonetheless be included within a single COREF tag. [In the future, it may be possible to capture the discontinuity explicitly by some special notation.] In the MUC-6 corpus, discontinuous noun phrases frequently appear in headlines, since the non-first lines of a headline are often marked with &quot;@&quot;, which is external to the preceding and subsequent text. An annotated example of a discontinuous markable is shown in the example below:</Paragraph> </Section> <Section position="3" start_page="1" end_page="1" type="sub_section"> <SectionTitle> 5.3 Exceptions: Articles </SectionTitle> <Paragraph position="0"> If the only difference between the head and the maximal noun phrase is the presence of an article -- the word &quot;the&quot;, &quot;a&quot;, or &quot;an&quot; at the beginning of the noun phrase -- the MIN need not be explicitly marked. (The scoring program will automatically strip leading articles before comparing strings.)</Paragraph> </Section> </Section> <Section position="6" start_page="1" end_page="2" type="metho"> <SectionTitle> 6. WHICH RELATIONSHIPS TO ANNOTATE </SectionTitle> <Paragraph position="0"/> <Section position="1" start_page="1" end_page="1" type="sub_section"> <SectionTitle> 6.1 Basic Coreference </SectionTitle> <Paragraph position="0"> The basic criterion for linking two markables is whether they are coreferential: whether they refer to the same object, set, activity, etc. It is not a requirement that one of the markables is &quot;semantically dependent&quot; on the other, or is an anaphoric phrase.</Paragraph> </Section> <Section position="2" start_page="1" end_page="1" type="sub_section"> <SectionTitle> 6.2 Bound Anaphors </SectionTitle> <Paragraph position="0"> We also make a coreference link between a &quot;bound anaphor&quot; and the noun phrase which binds it (even though one may argue that such elements are not coreferential in the usual sense). Thus we would link a quantified noun phrase and a pronoun dependent on that quantification: *Most computational linguists* prefer *their* own parsers.</Paragraph> <Paragraph position="1"> Note that a quantified noun phrase would also be linked to subsequent anaphors, outside the scope of quantification, through the usual relation of identity of coreference. Thus in the following text all three noun phrases would be linked: *Every TV network* reported *its* profits yesterday. *They* plan to release full quarterly statements tomorrow.</Paragraph> <Paragraph position="2"> By this rule, a pronoun in a relative clause which is bound to the head of the clause would get a coreference link to the entire NP. Thus, for every man who knows his own mind we would establish a coreference link between &quot;his&quot; and the entire noun phrase &quot;every man who knows his own mind&quot;: <COREF ID=&quot;1&quot; MIN=&quot;man&quot;>every man who knows <COREF ID=&quot;2&quot; REF=&quot;1&quot; TYPE=&quot;IDENT&quot;>his <COREF>own mind</COREF></Paragraph> </Section> <Section position="3" start_page="1" end_page="1" type="sub_section"> <SectionTitle> 6.3 Apposition </SectionTitle> <Paragraph position="0"> A typical use of an appositional phrase is to provide an alternative description or name for an object: Julius Caesar, the well-known emperor, Julius Caesar, a well-known emperor, the well known emperor, Julius Caesar, This identity of reference is to be represented by a coreference link between the appositional phrase, &quot;the well-known emperor&quot; and the ENTIRE noun phrase, &quot;Julius Caesar, the/a well-known emperor&quot;: <COREF ID=&quot;1&quot; MIN=&quot;Julius Caesar&quot;>Julius Caesar, <COREF ID=&quot;2&quot; REF=&quot;1&quot; MIN=&quot;emperor&quot; TYPE=&quot;IDENT&quot;> the/a well-known emperor,</COREF></COREF> The appositional phrase may be separated from the head by other modifiers. Thus Peter Holland, 45, deputy general manager,...</Paragraph> <Paragraph position="1"> becomes <COREF ID=&quot;1&quot; MIN=&quot;Peter Holland&quot;>Peter Holland, 45, <COREF ID=&quot;2&quot; REF=&quot;1&quot; TYPE=&quot;IDENT&quot; MIN=&quot;manager&quot;> deputy general manager,</COREF> </COREF> Appositional phrases are markable (and support the Descriptor slot in the Template Element task in the MUC-7 Information Extraction Task Definition) even when indefinite, e.g., Ms. Ima Head, a 10-year MUC veteran, San Diego, one of America's finest cities, An appositional phrase is also marked in the specifier relation, e.g., <COREF ID=&quot;1&quot; MIN=&quot;job&quot;>The job of <COREF ID=&quot;2&quot; REF=&quot;1&quot;>manager</COREF></ COREF> However, appositional phrases are NOT marked when they are negative: Ms. Ima Head, never a great MUC fan, or when there is only partial overlap of sets: The criminals, often legal immigrants, ...</Paragraph> <Paragraph position="2"> Appositional phrases are marked only when they constitute a separate noun phrase following the head. In written text, appositives are generally set off by commas; in transcripts of spoken language, the commas may well not be present because punctuation is generally not captured in text-to-speech transcription. There are cases where a construction that looks similar to an appositive but occurs within a single noun phrase as a title or modifier, e.g., *the real estate company* * Century 21* This kind of single noun construction is not considered markable. Thus, no coreference is marked in cases such as the following: the real estate company Century 21 the realtor Century 21 presidential advisor Joe Smarty</Paragraph> </Section> <Section position="4" start_page="1" end_page="1" type="sub_section"> <SectionTitle> Treasury Secretary Bucks </SectionTitle> <Paragraph position="0"> But the following phrase would have mark-up: *the job of *manager**</Paragraph> </Section> <Section position="5" start_page="1" end_page="2" type="sub_section"> <SectionTitle> 6.4 Predicate Nominals and Time-dependent Identity </SectionTitle> <Paragraph position="0"> Predicate nominals are also typically coreferential with the subject. Thus in the example *Bill Clinton* is *the President of the United States*.</Paragraph> <Paragraph position="1"> we would record a coreference link between &quot;Bill Clinton&quot; and &quot;the President of the United States&quot;. Coreference should NOT be recorded if the text only asserts the possibility of identity between two markables. In Phinneas Flounder may be the dumbest man who ever lived.</Paragraph> <Paragraph position="2"> Phinneas Flounder was almost the first president of the corporation.</Paragraph> <Paragraph position="3"> If elected, Phinneas Flounder would be the first Californian in the Oval Office. no coreference is to be recorded.</Paragraph> <Paragraph position="4"> We also allow coreference to be recorded when the predicate nominative is marked indefinite, e.g.,: *Mediation* is *a viable alternative to bankruptcy*.</Paragraph> <Paragraph position="5"> *Farm-debt mediation* is *one of the Farm Belt's success stories*.</Paragraph> <Paragraph position="6"> *ARPA program managers* are *nice people*.</Paragraph> <Paragraph position="7"> However, as with apposition, if there is possibility or a partial set overlap, no coreference is marked because there is no set coreference and no IDENT relation: Mediation is often a viable alternative to bankruptcy.</Paragraph> <Paragraph position="8"> Mediation may be a viable alternative to bankruptcy.</Paragraph> <Paragraph position="9"> Two markables should be recorded as coreferential if the text asserts them to be coreferential at ANY TIME. Thus Henry Higgins, who was formerly sales director for Sudsy Soaps, became president of Dreamy Detergents should be annotated as <COREF ID=&quot;1&quot; MIN=&quot;Henry Higgins&quot;>Henry Higgins, who was formerly <COREF ID=&quot;2&quot; MIN=&quot;director&quot; REF=&quot;1&quot; TYPE=&quot;IDENT&quot;>sales director for Sudsy Soaps,</COREF></COREF> became <COREF ID=&quot;3&quot; MIN=&quot;president&quot; REF=&quot;1&quot; TYPE=&quot;IDENT&quot;>president of Dreamy Detergents</COREF> This is one portion of the guidelines that will clearly need modification after a decision is made about enhancing the notation to distinguish time-dependent coreference links from other coreference links. But the distinction between the two types applies not only to predicate nominals but also to apposition, function-value and other construction types. Thus the phrase &quot;and Time-Dependent Entity&quot; should probably be removed from the title of section 6.4; a new section could cover the general issue of time-dependent coreference. Also, general info about any new type of notation should go in section 2, and the meaning of the new notation should be documented in section 6 (which may need a different section title).</Paragraph> <Paragraph position="10"> Even if the copula or inchoative verb is embedded, coreference should be marked, as in Dreamy Detergents named Henry Higgins to be president which should be annotated as Dreamy Detergents named <COREF ID=&quot;1&quot;>Henry Higgins</COREF> to be <COREF ID=&quot;2&quot; REF=&quot;1&quot; TYPE=&quot;IDENT&quot;>president</COREF> When the copula is clearly implied by the semantics of the verb, coreference should be marked. Expressions of equivalence involving the word &quot;as&quot; will also be marked. The NPs enclosed in asterisks in the following examples will be marked coreferential: Dreamy Detergents named *Henry Higgins* *president* *Henry Higgins* is considered *Sudsy Soap's best sales director* *Higgins* will serve as *president of Dreamy Detergents* Cases may arise where an intensional descriptor may apply to two distinct entities, e.g., the current president of a company and a previous president are mentioned. The conventions require that the extensional descriptions guide the mark-up process, and therefore that these two chains *NOT* be collapsed into a single chain: Both coreference chains contain the same intensional predicate, &quot;sales director for Sudsy Soaps&quot;, but these have different temporal extensional realizations. Although the occurrences of &quot;sales director&quot; are coreferential at the type level, the extensionally grounded chains take precedence, because it is critical to preserve independence of chains grounded in different extensions -- that is, to prevent obviously different individuals from ending up in the same IDENT coreference chain or equivalence class. Thus we &quot;cut&quot; the chain in the above example at the first &quot;type&quot; coreference that would cause collapsing (that is, that can point to a new extension), as in: <COREF ID=&quot;5&quot;>Fred</COREF> resigned as <COREF ID=&quot;6&quot; MIN=&quot;president&quot; REF=&quot;5&quot;>president of IBM</COREF>; next month, <COREF ID=&quot;7&quot;>the president</COREF> will be <COREF ID=&quot;8&quot; REF= &quot;7&quot;>Mary</COREF>.</Paragraph> </Section> <Section position="6" start_page="2" end_page="2" type="sub_section"> <SectionTitle> 6.5 Types and Tokens </SectionTitle> <Paragraph position="0"> The general principle for annotating coreference is that two markables are coreferential if they both refer to sets, and the sets are identical, or they both refer to types, and the types are identical. There are a number of problematic cases where one can argue whether something is a set or a type. There is no simple algorithm for determining the ontological category of a referent. There are, though, some useful rules. Most occurrences of bare plurals refer to types or kinds, not to sets. In ...*producers* don't like to see a hit wine increase in price... *Producers* have seen this market opening up and *they*'re now creating wines that appeal to these people.</Paragraph> <Paragraph position="1"> &quot;producers&quot;, &quot;Producers&quot;, and &quot;they&quot; refer to types and they all refer to the same type. Notice that if interpreted as referring to sets, they would not all refer to the same set. More properly, there is no reason to think they would corefer; not all the producers who have seen the market opening up have created new wines.</Paragraph> <Paragraph position="2"> Note that a type can be referred to by a bare plural, a definite singular NP (&quot;the tiger is fast becoming extinct&quot;) or a (bare) prenominal. In The action followed by one day an Intelogic announcement that it will retain an investment banker to explore alternatives &quot;to maximize *shareholder* value,&quot; including the possible sale of the company. Mr. Edelman declined to specify what prompted the recent moves, saying they are meant only to benefit *shareholders* when &quot;the company is on a roll.&quot; the two starred occurrences corefer to the type: shareholder (of Intelogic).</Paragraph> </Section> <Section position="7" start_page="2" end_page="2" type="sub_section"> <SectionTitle> 6.6 Functions and Values In </SectionTitle> <Paragraph position="0"> GM announced *its third quarter profit*. *It* was *$0.02*.</Paragraph> <Paragraph position="1"> all three starred phrases refer to an amount of money; they all refer to the same amount of money. Hence they are coreferential. The first phrase, in context, refers to that amount via referring to a function, say of companies and quarters of a year--or times. (In addition, the &quot;its&quot; in the first NP would be linked to GM.) In General Motors announced {their third quarter profit of *$0.02*}.</Paragraph> <Paragraph position="2"> the bracketed and starred phrases are coreferential. They refer to one and the same amount of money. Note that here, as in the case of apposition, the result is that a phrase is marked as being coreferential with a part of the phrase.</Paragraph> <Paragraph position="3"> In *The temperature* is *90*....The temperature is rising.</Paragraph> <Paragraph position="4"> the first occurrence of &quot;the temperature&quot; is an intensional expression referring to the value (extension) of the function at arguments (places, times) supplied by context. *The temperature* is coreferential with &quot;90&quot; which grounds it. In the second occurrence, &quot;the temperature&quot; refers to the function (indirectly, by way of referring to the derivative of the function). So it is not coreferential with the first occurrence or with &quot;90&quot;. In the sequence The temperature was 90 yesterday and has already reached 95 today. This sets a new record high. we have a different problem: we have two extensional descriptions (90, 95) for the temperature, and only a single occurrence of the intensional description &quot;temperature&quot;. In this case, &quot;temperature&quot; is coreferential with the extensional description occurring in the same clause (&quot;90&quot;). As a result, &quot;95&quot; is in its own coreference class, and we are not able to mark the fact that it too is a temperature. However, &quot;95&quot; is coreferential with &quot;This&quot; and &quot;a new record high&quot;. This is marked as follows: <COREF ID=&quot;4&quot;>The temperature</COREF> was <COREF ID=&quot;5&quot; REF=&quot;4&quot;>90</ COREF> yesterday and has already reached <COREF ID=&quot;6&quot;>95</COREF> today. <COREF ID=&quot;7&quot; REF=&quot;6&quot;>This</COREF> sets <COREF ID=&quot;8&quot; MIN=&quot;high&quot; REF=&quot;7&quot;>a new record high</COREF>.</Paragraph> <Paragraph position="5"> If both extensions are in the same clause, as in: The stock value rose from $8.05 to $9.15 The per share value of $8.05 rose to $9.15 at the end of trading.</Paragraph> <Paragraph position="6"> then the function takes on the most &quot;current&quot; value in its clause, e.g., *stock value* and *$9.15* are marked coreferential, and $8.05 is in its own class, not coreferential.</Paragraph> </Section> <Section position="8" start_page="2" end_page="2" type="sub_section"> <SectionTitle> 6.7 Metonymy </SectionTitle> <Paragraph position="0"> The pervasive phenomenon of metonymy raises a problem for Coreference relations. Do we annotate and recognize the relation before or after coercion? Here are some texts to consider: (1) *The White House* sent its health care proposal to Congress yesterday. Senator Dole said *the administration*'s bill had little chance of passing.</Paragraph> <Paragraph position="1"> (2) *Ford* announced a new product line yesterday. *Ford* spokesman John Smith said *they* will start manufacturing widgets.</Paragraph> <Paragraph position="2"> (3) I bought the New York Times this morning. I read that the editor of the New York Times is resigning. (4) *The United States* is a democracy. *The United States* has an area of 3.5 million square miles. We propose that coreference be determined with respect to coerced entities. Of course, this still leaves open the question as to the circumstances under which coercion is required. In (1) there is a coercion from the White House to the administration operating out of the White House, and that is IDENT with &quot;the administration&quot;; so &quot;White House&quot; and &quot;administration&quot; are IDENT. (Notice that there is also a question as to whether the adminstration's proposal is the same as its bill. This too requires a coercion of sorts.) In (2), while there might seem to be a coercion from Ford to a spokesman for Ford, we believe that such a coercion is not necessary, for it is plausible that corporations, as legal persons, can do many of the things that people can do--such as &quot;announce.&quot; They may have to do some or all such things through other agents, but many people do many things that way. And if Ford can announce, then it, through one of its spokesmen, can &quot;say&quot;. Believing that no coercion is required, we would mark as coreferential the first instance of &quot;Ford&quot;, the second instance of &quot;Ford&quot; (in the phrase &quot;Ford spokesman John Smith&quot;), and &quot;they&quot;, but would NOT mark the phrase &quot;Ford spokesman John Smith&quot; as coreferential with anything else in this passage. In (3) the first &quot;New York Times&quot; is coerced into a copy of the paper published by the New York Times and the second is coerced into the organization; so they are not IDENT. (4) is somewhat akin to (2). Countries are both geographical entities and governmental units. Thus, no coercion is necessary and the two starred occurrences are coreferential.</Paragraph> <Paragraph position="3"> In the absence of general principles, a body of such decisions will need to be developed to codify the rules for coercion and coreference. In cases where there has been no clear precedent, the answer keys for formal evaluations will need to mark coreference as optional.</Paragraph> </Section> </Section> <Section position="7" start_page="2" end_page="2" type="metho"> <SectionTitle> 7. BASIS OF JUDGMENT </SectionTitle> <Paragraph position="0"> The coreference judgments should be based on the intelligent reader's knowledge of the world resulting from his or her best understanding of the text. It should not be based on a theory of the structure of the text, or on a linguistic theory of how NPs are resolved, or on estimates of what the typical NLP system could do. This means that some relations will be impossible for current NLP systems to recover, but this is why the task will push the technology. The annotators should assume that they are typical intelligent readers.</Paragraph> <Paragraph position="1"> 8. SCORING AND THE ORDERING OF LINKS If three markables, A, B, and C, are coreferential, this relationship could be recorded in the key in several ways: for example, by a REF pointer in both B and C pointing to A, or by a REF pointer in B pointing to A and a REF pointer in C pointing to B. A similar range of variations is possible in a system response. The current scoring rules provide that any correct key, when compared to any correct response, will yield a 100% recall/100% precision score, independent of the way the coreference relation is encoded in the key by REF pointers.</Paragraph> </Section> class="xml-element"></Paper>