File Information
File: 05-lr/acl_arc_1_sum/cleansed_text/xml_by_section/metho/04/w04-2327_metho.xml
Size: 29,288 bytes
Last Modified: 2025-10-06 14:09:22
<?xml version="1.0" standalone="yes"?> <Paper uid="W04-2327"> <Title>The MATE/GNOME Proposals for Anaphoric Annotation, Revisited</Title> <Section position="3" start_page="0" end_page="0" type="metho"> <SectionTitle> 2 The MATE Proposals </SectionTitle> <Paragraph position="0"> The design of an annotation scheme involves a number of decisions: what has to be annotated, how, and how the annotation should be recorded (the markup scheme).</Paragraph> <Paragraph position="1"> One of the most important motivations behind the design of the MATE proposals for anaphoric annotation is the belief that given the variety of phenomena that go under the name of anaphora, and the variety of possible applications, there can be no such thing as a general-purpose anaphoric annotation instructions. On the other hand, we also believed that it is possible to design a general purpose markup scheme (and therefore, general-purpose tools) that could then be used in different ways for different projects. The approach taken in MATE was then to design a general markup scheme (the 'meta-scheme') and then to show its basic building blocks could be used to implement different types of anaphoric annotation, including some of the most popular schemes for 'coreference annotation,' such as the MUC scheme (MUCCS) (Hirschman, 1998), Passonneau's DRAMA scheme (1997) , and the scheme used for annotation of references to landmarks in the MapTask corpus. In this section we summarize the most distinctive features of the proposals resulting from this basic assumption. The full description of the MATE scheme is available from the MATE project pages at http://mate.nis.sdu.dk/.</Paragraph> <Section position="1" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 2.1 Coreference, Anaphora and Discourse Modeling </SectionTitle> <Paragraph position="0"> The MATE scheme differs from the best-known scheme for annotating 'coreference,' MUCCS (Hirschman, 1998) both in the conceptualization underlying the annotation (i.e., what type of information should be annotated) and in the way this information is marked up. MUCCS was designed to encode information deemed useful for a sub-task of information extraction, and the instructions provided to annotators were meant to ensure that all information provided by a text about a certain entity would be marked using a single device, the IDENT relation. As van Deemter and Kibble (2000) point out, however, the result is rather ad hoc; the IDENT relation as defined by the instructions doesn't capture any coherent definition of 'coreference'. (In fact, the very notion of 'reference' is rather difficult to formalize precisely.) The MATE proposals, by contrast, while still labeled as proposals for 'coreference annotation,' because the name has become a de facto standard as a result of the MUC initiative, are explicitly based on the DISCOURSE MODEL assumption adopted almost universally by linguists (computational and not) working on anaphora resolution and generation (Webber, 1979; Heim, 1982; Kamp and Reyle, 1993; Gundel et al., 1993). This is the hypothesis that interpreting a discourse involves building a shared discourse model containing DISCOURSE ENTITIES that may or may not 'refer' to specific objects in the world, as well as the relations between these entities. The type of annotation for which the MATE scheme was developed-and that we'll call here 'anaphoric annotation,'.1 is meant as a partial representation of the discourse model evoked by a text (hence, for example, the tag used for nominal expressions denoting discourse entities, <de> ).2</Paragraph> </Section> <Section position="2" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 2.2 The Markup Scheme </SectionTitle> <Paragraph position="0"> The design of the MATE workbench was strongly inspired by the concept of STANDOFF ANNOTATION developed for the reorganization of the MapTask. The main principle of standoff annotation is that each level of annotationfor example, syntactic annotation, dialogue act annotation, and anaphoric annotation-should be stored independently; in this way, annotators working on one level need not be concerned about the other levels of annotation, and can start immediately without having to wait for other annotation tasks to be completed. The separate levels of annotation are synchronized via a base file, to which the separate levels point using the HREF mechanism of XML.</Paragraph> <Paragraph position="1"> The markup scheme for anaphoric relations is the core aspect of the MATE proposals and its most distinctive aspect. As in the MUC scheme, it is as1van Deemter and Kibble (2000) give a stricly textual definition of 'anaphora' which is very distant from the common use of the term 'anaphora resolution' in computational linguistics, typically used to indicate the interpretation of (parts of) the meaning of an expression with respect to the discourse model.</Paragraph> <Paragraph position="2"> 2In fact, the use of the term 'coreference annotation' would not be completely misguided. van Deemter and Kibble (2000) assume the definition of 'reference' typically found in formal semantics, but in functional linguistics, the term 'referring expression' is used to indicate expressions that introduce new discourse entities in a discourse model or that denote an old one (see, e.g., (Gundel et al., 1993)).</Paragraph> <Paragraph position="3"> sumed that annotation of anaphoric information involves identifying MARKABLES (the text constituent that realize semantic objects that may enter in anaphoric relations), and marking up anaphoric relations between them.</Paragraph> <Paragraph position="4"> The main difference from MUCCS is that whereas in MUCCS anaphoric relations are annotated using an attribute of the markables, in the MATE markup schemefollowing the recommendations of the Text Encoding Initiative (Burnard and Sperberg-McQueen, 2002), and of Bruneseaux and Romary (1998)-the distinction between these two steps of annotation is mirrored by a distinction between two XML elements: <de> , used to indicate the markables, and <link> , used to mark information about anaphoric relations (or any other semantic relation).3 However, unlike in the TEI proposals, in the MATE markup scheme <link> elements are structured elements, containing one or more <anchor> element.</Paragraph> <Paragraph position="5"> The <link> element specifies the anaphoric expression (using XML'sHREFmechanism) and the relation between the anaphoric expression and its antecedent; whereas the <anchor> element specifies the antecedent, as in (1) where, for example, the first <link> elements encodes the information that the discourse entities realized by the NPs the engine E3 and it denote the same object.</Paragraph> <Paragraph position="6"> (1) coref.xml <de ID=&quot;de_01&quot;>we</de>'re gonna take <de ID=&quot;de_07&quot;> the engine E3 </de> and shove <de ID=&quot;de_08&quot;> it </de> over to <de ID=&quot;de_02&quot;>Corning</de>, hook <de ID=&quot;de_09&quot;> it </de> up to <de ID=&quot;de_03&quot;>the tanker car</de>...</Paragraph> <Paragraph position="8"> There were two main reasons for having <link> elements separated from the elements used to indicate markables. The first reason is that in this way <link> elements can be kept in a separate file from <de> elements, in keeping with the idea of standoff annotation. The second, and more important, reason is that in this way it is possible to annotate multiple anaphoric relations involving the same anaphoric expression without having multiple attributes for each markable.</Paragraph> <Paragraph position="9"> The reason why <link> elements may have more than one <anchor> element is to allow for the possibility to annotate ambiguities. For some types of applications, it may be a good idea not to ask annotators to decide upon the interpretation of ambiguous anaphoric expressions.</Paragraph> <Paragraph position="10"> 3It was assumed that the tags for 'coreference' annotation would be part of a special namespace, COREF-i.e., that the actual name of these tags are <coref:de> , <coref:link> , etc. We omit the namespace indication in this paper.</Paragraph> <Paragraph position="11"> In these cases, the multiple anchors mechanisms allows each of the possibilities to be marked by means of a separate <anchor> element. In (2a), for example, the pronun it in 15.16 could refer equally well to engine E3 or the tanker car. With the MATE mechanism, both antecedents can be annotated, as shown in (2b).</Paragraph> <Paragraph position="12"> (2) a. 15.12 : we're gonna take the engine E3 15.13 : and shove it over to Corning 15.14 : hook it up to the tanker car 15.15 : _and_ 15.16 : and send it back to Elmira b. coref.xml: 15.12 : we're gonna take <de ID=&quot;de_15&quot;>the engine E3</de> 15.13 : and shove <de ID=&quot;de_16&quot;> it </de> over to Corning 15.14 : hook <de ID=&quot;de_17&quot;>it</de> up to <de ID=&quot;de_18&quot;>the tanker car</de> 15.15 : _and_ 15.16 : and send <de ID=&quot;de_19&quot;>it</de> back to Elmira</Paragraph> </Section> <Section position="3" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 2.3 Instantiations of the Meta-Scheme </SectionTitle> <Paragraph position="0"> As said above, the markup elements just discussed were meant to be general enough to support different types of annotation. Three such examples were considered.</Paragraph> <Paragraph position="1"> The Core Scheme In the most basic type of coreference scheme, only anaphoric relations between NPs are considered, and only identity relations. Schemes of this type can be implemented by having just one anaphoric relation, IDENT. The remaining differences between the schemes have then mostly to do with the instructions to annotators-for example, which types of anaphoric relations to be considered as cases of 'identity' (see (van Deemter and Kibble, 2000) for some problems with the choices made in MUCCS). In the comments for the designers of a scheme, it was suggested that some of the cases marked as coreference in MUCCS, such as the relation between the temperature and 90 degrees in the temperature rose to 90 degrees before dropping to 70 degrees, would be best marked as function-value relations (viewing the temperature as a function from objects and time points into values, rather than an individual-denoting term).</Paragraph> <Paragraph position="2"> Extended Relations In DRAMA, a number of associative relations are considered, such as SUBSET or PART, together with instructions how to annotate them. This types of anaphora can be annotated in the MATE markup scheme using additional relations, as in (3), where the discourse entity realized by LES FUSEES QUI ONT BIEN VOLE' denotes a subset of the set denoted by discourse entity DE 88, LES MODELES DE FUSEES.</Paragraph> <Paragraph position="3"> It was pointed out, however, that the results of Poesio and Vieira (1998) indicated that this type of annotation could be highly unreliable.</Paragraph> <Paragraph position="4"> References to the Visual Situation A special <universe> element was suggested for MapTaskstyle annotations of references to visible objects. The <universe> element containing one <ue> element for each object in the visual scene; including such elements in an annotation makes it possible to use <link> elements to annotate references to such objects.4 Cases in which the participants to a conversation have different visual situations, as in the MapTask dialogues, can be handled by having separate universes, one for each participant to the conversation. In addition, a WHO-BELIEVES attribute of <link> elements was proposed to represent situations in which only one participant believes that a particular anaphoric relation holds, as in example (7) (Appendix A), where it's only the follower to believe that a gold mine refers to the same object as diamond mine.</Paragraph> </Section> <Section position="4" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 2.4 Instructions for Identifying Markables </SectionTitle> <Paragraph position="0"> Because the goal of the MATE annotation proposals was to provide a set of tools that could be used to implement a variety of options, rather than to identify a specific scheme appropriate for all applications, it didn't make sense to specify detailed instructions for annotation. However, a substantial effort was made to provide an exhaustive inventory of the options for identi4The <universe> mechanism is based on the notion of 'anchor' developed in Discourse Representation Theory (DRT), although simplified in a number of ways.</Paragraph> <Paragraph position="1"> fying markables that were available to the designers of a scheme for anaphoric annotation. These suggestions were in part derived from MUCCS and from Passonneau's DRAMA scheme, but a number of additional problems were considered as well.</Paragraph> <Paragraph position="2"> As in MUCCS, it was assumed that annotation of anaphora is best separated in two steps: first the markables (the text constituent that realize semantic objects that may enter in anaphoric relations) are agreed upon, then anaphoric relations between them are marked.</Paragraph> <Paragraph position="3"> Concerning markable identification, the main suggestions were to concentrate on anaphoric expressions realized as NPs and their antecedents; and to rely on the output of a parser as much as possible. But because of the assumption that only NPs evoking discourse entities should be considered, it was suggested that not all NP should be treated as markables: for example, it was recommended that NPs in post-verbal position in predicative clauses (such as a policeman in John is a policeman) should be excluded. This recommendation was later reconsidered (see below).</Paragraph> <Paragraph position="4"> One of the novel aspects of the MATE instructions was the concern for markable identification in languages other than English. One such issue was how to deal with incorporated clitics and empty subjects; the suggestion was to use a separate element, <seg> , to turn verbs into non-nominal markables, as in the following example: It was also proposed that the <seg> element could be used in more ambitious schemes as general mechanism for specifying non-nominal markables -e.g., in ellipsis, to indicate the antecedents of discourse deixis, etc.5</Paragraph> </Section> </Section> <Section position="4" start_page="0" end_page="0" type="metho"> <SectionTitle> 3 Work based on the MATE proposals </SectionTitle> <Paragraph position="0"> Ideas from the MATE 'scheme' have been adopted and tested both in annotation projects and by the developers of annotation tools. In this section we review some of these activities and summarize the conclusions concerning advantages and disadvantages of the MATE scheme that can be drawn from them.</Paragraph> <Paragraph position="1"> 5A second range of issues considered in the MATE scheme had to do with dialogue phenomena, such as non-contiguous elements; we will not consider these issues here.</Paragraph> <Section position="1" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 3.1 Annotation work related to the GNOME project </SectionTitle> <Paragraph position="0"> The most direct application of the ideas discussed above was found in the annotation work undertaken as part of the GNOME project. GNOME was concerned with the empirical investigation of the aspects of discourse that appear to affect generation, especially salience (Pearson et al., 2000; Poesio et al., 2000; Poesio and Di Eugenio, 2001; Poesio and Nissim, 2001; Poesio et al., 2004c).</Paragraph> <Paragraph position="1"> Particular attention was paid to the factors affecting the generation of pronouns (Pearson et al., 2000; Henschel et al., 2000), demonstratives (Poesio and Nygren-Modjeska, To appear) possessives (Poesio and Nissim, 2001) and definites in general (Poesio, 2004). These results, and the annotated corpus, were applied to the development of both symbolic and statistical natural language generation algorithms with the application of these empirical results to natural language generation, from sentence planning (Poesio, 2000a; Henschel et al., 2000; Cheng et al., 2001), to aggregation (Cheng, 2001) and text planning (Kibble and Power, 2000; Karamanis, 2003). The empirical side of the project involved both psychological experiments and corpus annotation, based on a scheme based on the MATE proposals, as well as on a detailed annotation manual (Poesio, 2000b), the reliability of whose instructions was tested by extensive experiments (Poesio, 2000a). More recently, the corpus has also been used to develop and evaluate anaphora resolution systems, with a special focus on the resolution of bridging references (Poesio, 2003; Poesio and Alexandrov-Kabadjov, 2004; Poesio et al., 2004b).</Paragraph> <Paragraph position="2"> The corpus The GNOME corpus currently includes texts from three domains, about 3000 NPs were annotated in each domain. The museum subcorpus consists of descriptions of museum objects, generally with an associated picture, and brief texts about the artists that produced them. The pharmaceutical subcorpus is a selection of leaflets providing the patients with legally mandatory information about their medicine.</Paragraph> <Paragraph position="3"> Several layers of information were annotated, including layout in the case of text and rhetorical structure in the case of tutorial dialogues, sentences and potential utterances, noun phrases, a variety of attributes of the objects denoted by noun phrases,6 and anaphoric relation.</Paragraph> <Paragraph position="4"> We concentrate here on anaphoric information, and refer the reader to the manual for the other types of annotation.</Paragraph> <Paragraph position="5"> Markup scheme The markup scheme for markables and anaphoric relations adopted in GNOME follows very 6E.g., whether an NP denoted generically or not; whether it denoted an animate or inanimate entity, as well as other ontological properties; and whether it denoted a discourse entity, a quantifier, or a predicate. In the case of a discourse entity, we also annotated whether it denoted an atom, a set, or a mass term; and whether it denoted uniquely or not.</Paragraph> <Paragraph position="6"> closely that proposed in MATE, except that the <de> element was renamed <ne> (since all NPs were marked), and the <link> element was renamed <ante> . More substantial differences are the decision not to use standoff, and the introduction of new elements necessary for the study of salience, such as elements that could be used to investigate the notion of UTTERANCE used in Centering (Poesio et al., 2004c).</Paragraph> <Paragraph position="7"> Although standoff is a clear improvement over including all annotation levels in a single file, our own experiences during the creation of the GNOME corpus being further proof of this, it's only really possible when tools are available both to create the annotation and-cruciallylater to 'knit back' the separate levels when needed. As neither the MATE workbench nor any other tools based on standoff were available by the time the GNOME annotation started,7 in GNOME we didn't use standoff, but integrated all levels of annotation in one file; an Emacs mode was developed for the annotation. This decision made it very easy to use the annotated corpus for a number of studies, but did resulted in a number of problems, the main among which were that the annotators had to be very careful not to damage other annotations; that annotators working on one level were occasionally confused by annotations for other levels; and that the annotation work had to be organized in a careful sequential way even for levels that could have been annotated independently.</Paragraph> <Paragraph position="8"> The main new aspect of the markup scheme, especially as far as our studies of salience were concerned, are the elements used to annotate potential utterances in the sense of Centering (Grosz et al., 1995). In order not to prejudge the answer to the question of which text constituents are best viewed as utterances, we used a 'generic' element called <unit> to mark up finite and non-finite clauses, but also parentheticals and appositions, elements of bulleted lists, etc.</Paragraph> <Paragraph position="9"> The following example illustrates both the use of anaphoric annotation in the released MATE workbench.</Paragraph> <Paragraph position="10"> Bridging References Apart from the basic anaphoric relations of identity, in GNOME we were concerned with bridging references, hence our annotation scheme incorporated aspects of the 'Extended Relations' and the 'MapTask' instantiations of the MATE meta-scheme.</Paragraph> <Paragraph position="11"> One of our aims was to continue the work on bridging references annotation and interpretation in (Poesio and Vieira, 1998), which showed that marking up bridging references is quite hard. In addition, work such as (Sidner, 1979; Strube and Hahn, 1999) suggested that indirect realization can play a crucial role in maintaining the CB. After testing a few types of associative reference (Hawkins, 1978), we decided to annotate only three non-identity relations, as well as identity. These relations are a subset of those proposed in the 'extended relations' version of the MATE scheme: set membership (ELEMENT), subset (SUBSET), and 'generalized possession' (POSS), which includes both part-of relations and ownership relations. null Coder manual Perhaps the most important aspects of the annotation work in GNOME are the development of detailed instructions for annotators and the reliability experiments testing several aspects of the scheme, particularly the annotation of bridging references.</Paragraph> <Paragraph position="12"> The identification of sentences, units and markables was done entirely by hand, without encountering particular problems. (The Emacs mode, an extension of SGMLmode, provides some support for introducing new elements, marking regions, and attribute editing, as well as anaphoric annotation.) Unlike in MATE, all NPs were tagged as <ne> . The instructions for <unit> s were based on Marcu's proposals for discourse units annotation (Marcu, 1999). All attributes of sentences, <unit> s and <ne> s in the final version of the scheme, including DEIX, can be annotated reliably.</Paragraph> <Paragraph position="13"> In order to achieve reliability on anaphoric annotation, the range of anaphoric phenomena considered was restricted in many ways. Apart from marking a limited number of associative relations, the annotators only marked relations between objects realized by noun phrases and not, for example, anaphoric references to actions, events or propositions implicitly introduced by clauses or sentences. We also gave strict instructions to our annotators concerning how much to mark. They were told to mark all identity relations, but to mark associative relations only if either (i) no IDENT relation could be marked for the anaphoric expression, or (ii) an IDENT relation with an entity not mentioned in the previous <unit> . Furthermore, preferences were specified, e.g., for appositions: for example, in Francois, the Dauphin, the embedding NP would be chosen as an antecedent of subsequent anaphoric references, rather than the NP in appositive position.</Paragraph> <Paragraph position="14"> We found a reasonable, although by no means perfect, agreement on identity relations. In a typical analysis (two annotators looking at the anaphoric relations between 200 NPs) we observed no real disagreements; 79.4% of these relations were marked up by both annotators; 12.8% by only one of them; and in 7.7% of the cases, one of the annotators marked up a closer antecedent than the other.</Paragraph> <Paragraph position="15"> Limiting the relations did limit the disagreements among annotators on associative relations (only 4.8% of the relations are actually marked differently) but only 22% of bridging references were marked in the same way by both annotators; 73.17% of relations are marked by only one or the other annotator. Reaching agreement on this information involved several discussions between annotators and more than one pass over the corpus (Poesio, 2000a).</Paragraph> </Section> <Section position="2" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 3.2 Annotation tools </SectionTitle> <Paragraph position="0"> Although no annotation tool implementing the MATE or GNOME schemes as described exists, in the years after the development of the MATE guidelines tools supporting XML standoff annotation for coreference have appeared, including MMAX from EML (M&quot;uller and Strube, 2003) and the Annotator from ILSP. Although the format used for storing anaphoric information by these tools is not entirely satisfactory, the files they produce can be easily converted into MATE format.</Paragraph> <Paragraph position="1"> MMAX, for example, is based on a simplified stand-off format, in which three main files are maintained for each annotated file in the corpus: a base file containing the words, a file identifying sentences, and a file identifying markables. Anaphoric information is stored as attributes of the markables. Two special attributes are used for this purpose, and recognized by MMAX: the MEMBER attribute, used to indicate membership in a coreference chain (a coreference equivalence class), and the POINTER attribute, used to mark up to one associative anaphoric relation for each anaphoric expression. We discuss the use of MMAX in the VENEX project below.</Paragraph> </Section> <Section position="3" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 3.3 The VENEX Corpus </SectionTitle> <Paragraph position="0"> The VENEX corpus is an anaphorically annotated corpus of Italian being created in a joint project between the Universit'a di Venezia and the University of Essex. The corpus includes both texts (newspaper articles) and dialogues (an Italian version of the MapTask corpus). This project widened our experiences of annotation with the MATE scheme in a number of respects. First of all, a number of proposals contained in the MATE guidelines but not relevant for GNOME, including the suggestions for dealing with misunderstandings and for incorporated anaphoric expressions such as clitics, were tested. Secondly, in this project we are attempting to identify markables automatically as far a possible, and data are stored in a standoff format, using a modern annotation tool (MMAX) for the annotation.</Paragraph> <Paragraph position="1"> Markup Scheme As MMAX doesn't support <link> elements, and anaphoric information is stored with markables, it is necessary to use markable attributes to represent information that would have been encoded as part of the links. We used a separate attribute to specify the type of associative relation used by POINTER attribute, and a SPACE attribute to encode the information stored in the WHO-BELIEVES attribute of links (see below). In addition, only one MEMBER and POINTER attributes can be specified for each markable.</Paragraph> <Paragraph position="2"> This latter limitation wasn't much of a problem, given that the annotation instructions used in VENEX are derived from those developed for GNOME and also attempt to limit annotators to mark at most one identity and one bridging relation for each anaphoric expression. The separation of attributes of links proved, however, a problem, as annotators often forget to annotate one or the other.</Paragraph> <Paragraph position="3"> An additional problem is that the version of MMAX we used (0.92) only allows for one type of markable, meaning that <unit> elements could not be annotated, and instead of using separate <ne> and <seg> elements for nominal and non-nominal markables, a single markable had to be used (see below).8 Misunderstandings The MapTask part of the VENEX corpus contains numerous examples like (7), where the differences between Giver and Follower map lead to one participant believing that two objects are anaphorically related, while the other participant either is not aware of this or doesn't believe this to be the case. We found that after a few iterations of training, our annotators were able to handle these cases properly (a more formal evaluation is underway; we hope to report the results at the meeting). Again, the only problems were caused by the fact that these attributes had to be added to markables, which sometimes led to annotators forgetting to set them. (This was only required in case the default, that an anaphoric relation was in the common ground of both participants, didn't hold.)</Paragraph> </Section> </Section> class="xml-element"></Paper>