<?xml version="1.0" standalone="yes"?>
<Paper uid="J05-2005">
  <Title>Representing Discourse Coherence: A Corpus-Based Study</Title>
  <Section position="2" start_page="0" end_page="467" type="abstr">
    <SectionTitle>
1. Introduction
</SectionTitle>
    <Paragraph position="0"> An important component of natural language discourse understanding and production is having a representation of discourse structure. A coherently structured discourse here is assumed to be a collection of sentences that are in some relation to each other. This article aims to present a set of discourse structure relations that are easy to code and to develop criteria for an appropriate data structure for representing these relations.</Paragraph>
    <Paragraph position="1"> There have been two kinds of approaches to defining and representing discourse structure and coherence relations. These approaches differ with respect to what kinds of discourse structure they are intended to represent. Some accounts aim to represent the intentional-level structure of a discourse; in these accounts, coherence relations reflect how the role played by one discourse segment with respect to the interlocutors' intentions relates to the role played by another segment (e.g., Grosz and Sidner 1986). Other accounts aim to represent the informational structure of a discourse; in these accounts, coherence relations reflect how the meaning conveyed by one discourse segment relates to the meaning conveyed by another discourse segment (e.g., Hobbs 1985; Marcu 2000; Webber et al. 1999). Furthermore, accounts of discourse structure vary greatly with respect to how many discourse relations they assume, ranging from 2 (Grosz and Sidner 1986) to over 400 different coherence relations (reported in Hovy and  [?] Computer Laboratory and Genetics Department, Cambridge, CB3 0FD, U.K.</Paragraph>
    <Paragraph position="2"> E-mail: Florian.Wolf@cl.cam.ac.uk [?][?] Department of Brain and Cognitive Sciences, Cambridge, MA 02139. E-mail: egibson@mit.edu. Submission received: 15th June 2004; Revised submission received: 5th September 2004; Accepted for publication: 23rd October 2004 (c) 2005 Association for Computational Linguistics  Computational Linguistics Volume 31, Number 2 Maier [1995]). However, Hovy and Maier (1995) argue that, at least for informational-level accounts, taxonomies with more relations represent subtypes of taxonomies with fewer relations. This means that different informational-level-based taxonomies can be compatible with each other; they differ with respect to how detailed or fine-grained a manner they represent informational structures of texts. Going beyond the question of how different informational-level accounts can be compatible with each other, Moser and Moore (1996) discuss the compatibility of rhetorical structure theory (RST) (Mann and Thompson 1988) with the theory of Grosz and Sidner (1986). However, note that Moser and Moore (1996) focus on the question of how compatible the claims are that Mann and Thompson (1988) and Grosz and Sidner (1986) make about intentional-level discourse structure.</Paragraph>
    <Paragraph position="3"> In this article, we aim to develop an easy-to-code representation of informational relations that hold between sentences or other nonoverlapping segments in a discourse monologue. We describe an account with a small number of relations in order to achieve more generalizable representations of discourse structures; however, the number is not so small that informational structures that we are interested in are obscured. The goal of the research presented is not to encode intentional relations in texts. We consider annotating intentional relations too difficult to implement in practice at this time. Note that we do not claim that intentional-level structure of discourse is not relevant to a full account of discourse coherence; it just is not the focus of this article.</Paragraph>
    <Paragraph position="4"> The next section describes in detail the set of coherence relations we use, which are mostly based on Hobbs (1985). We try to make as few a priori theoretical assumptions about representational data structures as possible. These assumptions are outlined in the next section. Importantly, however, we do not assume a tree data structure to represent discourse coherence structures. In fact, a major result of this article is that trees do not seem adequate to represent discourse structures.</Paragraph>
    <Paragraph position="5"> This article is organized as follows. Section 2 describes the procedure we used to collect a database of 135 texts annotated with coherence relations. Section 3 describes in detail the descriptional inadequacy of tree structures for representing discourse coherence, and Section 4 provides statistical evidence from our database that supports this claim. Section 5 offers some concluding remarks.</Paragraph>
    <Paragraph position="6"> 2. Collecting a Database of Texts Annotated with Coherence Relations This section describes (1) how we defined discourse segments, (2) which coherence relations we used to connect discourse segments, and (3) how the annotation procedure worked.</Paragraph>
    <Section position="1" start_page="0" end_page="251" type="sub_section">
      <SectionTitle>
2.1 Discourse Segments
</SectionTitle>
      <Paragraph position="0"> There is agreement that discourse segments should be nonoverlapping spans of text.</Paragraph>
      <Paragraph position="1"> However, there is disagreement in the literature about how to define discourse segments (cf. the discussion in Marcu [2000]). Whereas some argue that discourse segments should be prosodic units (Hirschberg and Nakatani 1996), others argue for intentional units (Grosz and Sidner 1986), phrasal units (Lascarides and Asher 1993; Longacre 1983; Webber et al. 1999), or sentences (Hobbs 1985).</Paragraph>
      <Paragraph position="2"> For our database, we mostly adopted a clause-unit-based definition of discourse segments. We chose this method of segmenting discourse because it was easy to use.</Paragraph>
      <Paragraph position="3">  Wolf and Gibson Representing Discourse Coherence Table 1 Contentful conjunctions used to illustrate coherence relations.</Paragraph>
      <Paragraph position="4"> Cause-effect because; and so Violated expectation although; but; while Condition if . . . (then); as long as; while Similarity and; (and) similarly Contrast by contrast; but Temporal sequence (and)then;first,second,...;before;after;while Attribution accordingto...;...said;claimthat...;maintainthat...;statedthat... Example for example; for instance Elaboration also; furthermore; in addition; note (furthermore) that; (for, in, on, against, with,...)which;who;(for,in,on,against,with,...)whom Generalization in general However, we also assumed that contentful coordinating and subordinating conjunctions (cf. Table 1) can delimit discourse segments.</Paragraph>
      <Paragraph position="5"> Note that we did not classify and as delimiting discourse segments if it was used to conjoin nouns in a conjoined noun phrase, like dairy plants and dealers in example (1) (from wsj 0306; Wall Street Journal 1989 corpus [Harman and Liberman 1993]) or if it  was used to conjoin verbs in a conjoined verb phrase, like snowed and rained in example (2) (constructed): (1) Milk sold to the nation's dairy plants and dealers averaged $14.50 for each hundred pounds.</Paragraph>
      <Paragraph position="6"> (2) It snowed and rained all day long.</Paragraph>
      <Paragraph position="7">  We classified periods, semicolons, and commas as delimiting discourse segments. However, in cases like example (3) (constructed), in which they conjoin a complex noun phrase, commas were not classified as delimiting discourse segments. (3) John bought bananas, apples, and strawberries.</Paragraph>
      <Paragraph position="8"> We furthermore treated attributions (John said that . . .) as discourse segments. This was empirically motivated. The texts used here were taken from news corpora, and there, attributions can be important carriers of coherence structures. For instance, consider a case in which some source A and some source B both comment on some event X.It should be possible to distinguish between a situation in which source A and source B make basically the same statement about event X and a situation in which source A and source B make contrasting comments about event X. Note, however, that we treated cases like example (4) (constructed) as one discourse segment and not as two separate ones ( ...citedand transaction costs . . .). We separated attributions only if the attributed material was a complementizer phrase, a sentence, or a group of sentences. This is not the case in example (4): The attributed material is a complex NP (transaction costs from its 1988 recapitalization).</Paragraph>
      <Paragraph position="9">  (4) The restaurant operator cited transaction costs from its 1988 recapitalization.  Computational Linguistics Volume 31, Number 2</Paragraph>
    </Section>
    <Section position="2" start_page="251" end_page="251" type="sub_section">
      <SectionTitle>
2.2 Discourse Segment Groupings
</SectionTitle>
      <Paragraph position="0"> Adjacent discourse segments could, in our approach, be grouped together. For example, discourse segments were grouped if they all stated something that could be attributed to the same source (cf. section 2.3 for a definition of attribution coherence relations).</Paragraph>
      <Paragraph position="1"> Furthermore, discourse segments were grouped if they were topically related. For example, if a text discussed inventions in information technology, there could be groups of a few discourse segments each talking about inventions by specific companies. There might also be subgroups, consisting of several discourse segments each, talking about specific inventions at specific companies. Thus, marking groups could determine a partially hierarchical structure for the text.</Paragraph>
      <Paragraph position="2"> Other examples of discourse segment groupings included cases in which several discourse segments described an event or a group of events that all occurred before another event or another group of events described by another (group of) discourse segments. In those cases, what was described by a group of discourse segments was in a temporal sequence relation with what was described by another (group of) discourse segments (cf. section 2.3 for a definition of temporal-sequence coherence relations). Note furthermore that in cases in which one topic required one grouping and a following topic required a grouping that was different from the first grouping, both groupings were annotated.</Paragraph>
      <Paragraph position="3"> Unlike approaches such as the TextTiling algorithm (Hearst 1997), ours allowed partially overlapping groups of discourse segments. The idea behind this option was to allow groupings of discourse segments in which a transition discourse segment belonged to the previous as well as the following group. However, the option was not used by the annotators (i.e., in our database of 135 hand-annotated texts, there were no instances of partially overlapping discourse segment groups).</Paragraph>
    </Section>
    <Section position="3" start_page="251" end_page="255" type="sub_section">
      <SectionTitle>
2.3 Coherence Relations
</SectionTitle>
      <Paragraph position="0"> As pointed out in section 1, we aim to develop a representation of informational relations between discourse segments. Note one difference between schema-based approaches (McKeown 1985) and coherence relations as we used them: Whereas schemas are instantiated from information contained in a knowledge base, coherence relations as we used them do not make (direct) reference to a knowledge base.</Paragraph>
      <Paragraph position="1"> There are a number of different informational coherence relations, dating back, in their basic definitions, to Hume, Plato, and Aristotle (cf. Hobbs 1985; Hobbs et al. 1993; Kehler 2002). The coherence relations we used are mostly based on Hobbs (1985); below we describe each coherence relation we used and note any differences between ours and Hobbs's (1985) set of coherence relations (cf. Table 2 for an overview of how our set of coherence relations relates to the set of coherence relations in Hobbs [1985]).</Paragraph>
      <Paragraph position="2"> The kinds of coherence relations we used include cause-effect relations, as in example (5) (constructed), in which discourse segment 1 states the cause for the effect that is stated in discourse segment 2:  (5) Cause-effect 1. There was bad weather at the airport 2. and so our flight got delayed.</Paragraph>
      <Paragraph position="3">  Our cause-effect relation subsumed the cause as well as the explanation relation in Hobbs (1985). A cause relation holds if a discourse segment stating a cause occurs  Wolf and Gibson Representing Discourse Coherence before a discourse segment stating an effect; an explanation relation holds if a discourse segment stating an effect occurs before a discourse segment stating a cause. We encoded this difference by adding a direction to the cause-effect relation. In a graph, this can be represented by a directed arc going from cause to effect.</Paragraph>
      <Paragraph position="4"> Another kind of causal relation is condition. Hobbs (1985) does not distinguish condition relations from either cause or explanation relations. However, we felt that it might be important to distinguish between a causal relation describing an actual causal event (cause-effect, cf. above), on the one hand, and a causal relation describing a possible causal event (condition, cf. below), on the other hand. In example (6) (constructed), discourse segment 2 states an event that will take place if the event described by discourse segment 1 also takes place:  (6) Condition 1. If the new software works, 2. everyone should be happy.</Paragraph>
      <Paragraph position="5"> In a third type of causal relation, the violated expectation relation (also violated expectation in Hobbs [1985]), a causal relation between two discourse segments that normally would be present is absent. In example (7) (constructed), discourse segment 1 normally would be a cause for everyone's being happy; this expectation is violated by what is stated by discourse segment 2: (7) Violated expectation 1. The new software worked great, 2. but nobody was happy.</Paragraph>
      <Paragraph position="6">  Other possible coherence relations include similarity (parallel in Hobbs [1985]) or contrast (also contrast in Hobbs [1985]) relations, in which similarities or contrasts are determined between corresponding sets of entities or events, such as between discourse segments 1 and 2 in example (8) (constructed) and discourse segments 1 and 2 in  example (9) (constructed), respectively: (8) Similarity 1. The first flight to Frankfurt this morning was delayed.</Paragraph>
      <Paragraph position="7"> 2. The second flight arrived late as well.</Paragraph>
      <Paragraph position="8"> (9) Contrast 1. The first flight to Frankfurt this morning was delayed.</Paragraph>
      <Paragraph position="9"> 2. The second flight arrived on time.</Paragraph>
      <Paragraph position="10"> Discourse segments might also elaborate (also elaboration in Hobbs [1985]) on other sentences, as in example (10) (constructed), in which discourse segment 2 elaborates on discourse segment 1: (10) Elaboration 1. A probe to Mars was launched from the Ukraine this week.</Paragraph>
      <Paragraph position="11"> 2. The European-built &amp;quot;Mars Express&amp;quot; is scheduled to reach Mars by late December.</Paragraph>
      <Paragraph position="12">  Discourse segments can provide examples for what is stated by another discourse segment. In example (11) (constructed), discourse segment 2 states an example  Computational Linguistics Volume 31, Number 2 (exemplification in Hobbs [1985]) for what is stated in discourse segment 1:  (11) Example 1. There have been many previous missions to Mars.</Paragraph>
      <Paragraph position="13"> 2. A famous example is the Pathfinder mission.</Paragraph>
      <Paragraph position="14">  Hobbs (1985) also includes an evaluation relation, as in example (12) (from Hobbs [1985]), in which discourse segment 2 states an evaluation of what is stated in discourse segment 1. We decided to call such relations elaborations, since we found it too difficult in practice to reliably distinguish elaborations from evaluations (according to our annotation scheme, in example (12), what is stated in discourse segment 2 elaborates on what is stated in discourse segment 1):  (12) Elaboration (labeled as evaluation in Hobbs [1985]) 1. (A story.) 2. It was funny at the time.</Paragraph>
      <Paragraph position="15"> Unlike Hobbs (1985), we did not have a separate background relation as in example (13) (modified from Hobbs [1985]), in which what is stated in discourse segment 1 provides the background for what is stated in discourse segment 2. As with the evaluation relation, we found the background relation too difficult to reliably distinguish from elaboration relations (according to our annotation scheme, in example (13), what is stated in discourse segment 1 elaborates on what is stated in discourse segment 2): (13) Elaboration (labeled as background in Hobbs [1985]) 1. T is the pointer to the root of a binary tree.</Paragraph>
      <Paragraph position="16"> 2. Initialize T.</Paragraph>
      <Paragraph position="17"> In a generalization relation, as in example (14) (constructed), one discourse segment (here discourse segment 2) states a generalization for what is stated by another discourse segment (here discourse segment 1): (14) Generalization 1. Two missions to Mars in 1999 failed.</Paragraph>
      <Paragraph position="18"> 2. There are many missions to Mars that have failed.</Paragraph>
      <Paragraph position="19"> Furthermore, discourse segments can be in an attribution relation, as in example (15) (constructed), in which discourse segment 1 states the source of what is stated in discourse segment 2 (cf. [Bergler 1991] for a more detailed semantic analysis of attribution relations): (15) Attribution 1. John said that 2. the weather would be nice tomorrow.</Paragraph>
      <Paragraph position="20">  Hobbs (1985) does not include an attribution relation. However, we decided to include attribution as a relation because, as pointed out in section 2.1, the texts we annotated are taken from news corpora. There, attributions can be important carriers of coherence structures.</Paragraph>
      <Paragraph position="21">  Wolf and Gibson Representing Discourse Coherence In a temporal sequence relation, as in example (16) (constructed), one discourse segment (here discourse segment 1) states an event that takes place before another event stated by the other discourse segment (here discourse segment 2):  (16) Temporal Sequence 1. First, John went grocery shopping.</Paragraph>
      <Paragraph position="22"> 2. Then he disappeared in a liquor store.</Paragraph>
      <Paragraph position="23">  In contrast to cause-effect relations, there is no causal relation between the events described by the two discourse segments. The temporal sequence relation is equivalent to the occasion relation in Hobbs (1985).</Paragraph>
      <Paragraph position="24"> The same relation, illustrated by example (17) (constructed), is an epiphenomenon of assuming contiguous distinct elements of text (Hobbs [1985] does not include a same relation). A same relation holds if a subject NP is separated from its predicate by an intervening discourse segment. For instance, in example (17), discourse segment 1 is the subject NP of a predicate in discourse segment 3, and so there is a same relation between discourse segments 1 and 3; discourse segment 1 is the first and discourse segment 3 is the second segment of what is actually one single discourse segment, separated by the intervening discourse segment 2, which is in an attribution relation with discourse segment 1 (and therefore also with discourse segment 3, since discourse segments 1 and  3 are actually one single discourse segment): (17) Same 1. The economy, 2. according to some analysts, 3. is expected to improve by early next year.</Paragraph>
      <Paragraph position="25">  Table 2 provides an overview of how our set of coherence relations relates to the set of coherence relations in Hobbs (1985).</Paragraph>
      <Paragraph position="26"> We distinguish between asymmetrical or directed relations, on the one hand, and symmetrical or undirected relations, on the other hand (Mann and Thompson 1988; Marcu 2000). Cause-effect, condition, violated expectation, elaboration, example, generalization, attribution,andtemporal sequence are asymmetrical or directed relations, whereas similarity, contrast,andsame are symmetrical or undirected relations. In asymmetrical or directed relations, the directions of relations are as follows: a114 Cause-effect: from the discourse segment stating the cause to the discourse segment stating the effect a114 Condition: from the discourse segment stating the condition to the discourse segment stating the consequence a114 Violated expectation: from the discourse segment stating the cause to the discourse segment describing the absent effect a114 Elaboration: from the elaborating discourse segment to the elaborated discourse segment a114 Example: from the discourse segment stating the example to the discourse segment stating the exemplified a114 Generalization: from the discourse segment stating the special case to the discourse segment stating the general case</Paragraph>
    </Section>
    <Section position="4" start_page="255" end_page="256" type="sub_section">
      <SectionTitle>
Evaluation Elaboration
Background Elaboration
</SectionTitle>
      <Paragraph position="0"> Exemplification: example stated first, then Example general case; directionality indicated by directed arcs in a coherence graph Exemplification: general case stated first, then Generalization example; directionality indicated by directed arcs in a coherence graph  Attribution: from the discourse segment stating the source to the attributed statement a114 Temporal sequence: from the discourse segment stating the event that happened first to the discourse segment stating the event that happened second This definition of directionality is related to Mann and Thompson's (1988) notion of nucleus and satellite nodes (where the nodes can represent [groups of] discourse segments): For asymmetrical or directed relations, the directionality is from satellite to nucleus node; by contrast, symmetrical or undirected relations hold between two nucleus nodes.</Paragraph>
      <Paragraph position="1"> Note also that in our annotation project, we decided to annotate a coherence relation either if there was a coherence relation between the complete content of two discourse segments, or if there was a relation between parts of the content of two discourse segments. Consider the following example (from ap890104-0003; AP Newswire corpus;  [ in enacting the accord for the independence of Namibia ] 2. for which SWAPO has fought many years, For this example we would annotate an elaboration relation from discourse segment 2 to discourse segment 1 (discourse segment 2 provides additional details about the accord  Wolf and Gibson Representing Discourse Coherence mentioned in discourse segment 1), although the relation actually only holds between discourse segment 2 and the second part of discourse segment 1, indicated by brackets. Although it is beyond the scope of the current project, future research should investigate annotations with discourse segmentations that allow annotating relations only between parts of discourse segments that are responsible for a coherence relation. For example, consider example (19) (from ap890104-0003; AP Newswire corpus [Harman and Liberman 1993]), in which brackets indicate how more-fine-grained discourse segments might be marked:  [ has fought many years, ] 2. referring to the acronym of the South-West African Peoples  Organization nationalist movement.</Paragraph>
      <Paragraph position="2"> In our current project, we annotated an elaboration relation from discourse segment 2 to discourse segment 1 (discourse segment 2 provides additional details, the full name, for SWAPO, which is mentioned in discourse segment 1). A future, more detailed, annotation of coherence relations could then annotate this elaboration relation to hold only between discourse segment 2 and the word SWAPO in discourse segment 1.</Paragraph>
    </Section>
    <Section position="5" start_page="256" end_page="258" type="sub_section">
      <SectionTitle>
2.4 Coding Procedure
</SectionTitle>
      <Paragraph position="0"> To code the coherence relations of a text, we used a procedure consisting of three steps.</Paragraph>
      <Paragraph position="1"> In the first step, a text was segmented into discourse segments (cf. section 2.1).</Paragraph>
      <Paragraph position="2"> In the second step, adjacent discourse segments that were topically related were grouped together. The criteria for this step are described in section 2.2.</Paragraph>
      <Paragraph position="3"> In the third step, coherence relations (cf. section 2.3) were determined between discourse segments and groups of discourse segments. Each previously unconnected (group of) discourse segment(s) was tested to see whether it connected to any of the (groups of) discourse segments that had already been connected to the already existing representation of discourse structure.</Paragraph>
      <Paragraph position="4"> In order to help determine the coherence relation between (groups of) discourse segments, the annotators judged which, if any, of the contentful coordinating conjunctions in Table 1 resulted, when used, in the most acceptable passage (cf. Hobbs 1985; Kehler 2002). If using a contentful conjunction to connect (groups of) discourse segments resulted in an acceptable passage, this was used as evidence that the coherence relation corresponding to the mentally inserted contentful conjunction held between the (groups of) discourse segments under consideration. This mental exercise was done only if there was not already a contentful coordinating conjunction that disambiguated the coherence relation.</Paragraph>
      <Paragraph position="5"> The following list (which was also used by the annotators to guide them in their task) shows in more detail how the annotations were carried out:  1. Segment the text into discourse segments: (a) Insert segment boundaries at every period that marks a sentence boundary (i.e., not at periods such as those in Mrs. or Dr.).</Paragraph>
      <Paragraph position="6"> (b) Insert segment boundaries at every semicolon and colon that marks a sentence or clause boundary.</Paragraph>
      <Paragraph position="7"> (c) Insert segment boundaries at every comma that marks a sentence or clause boundary; do not insert segment boundaries at commas that conjoin complex noun or verb phrases.</Paragraph>
      <Paragraph position="8">  Computational Linguistics Volume 31, Number 2 (d) Insert segment boundaries at every quotation mark, if there is not already a segment boundary based on steps (a)-(c).</Paragraph>
      <Paragraph position="9"> (e) Insert segment boundaries at the contentful coordinating conjunctions listed in Table 1, if there is not already a segment boundary based on steps (a)-(d). For and, do not insert a segment boundary if it is used to conjoin verbs or nouns in a conjoined verb  or noun phrase.</Paragraph>
      <Paragraph position="10"> 2. Generate groupings of related discourse segments: (a) Group contiguous discourse segments that are enclosed by pairs of quotation marks.</Paragraph>
      <Paragraph position="11"> (b) Group contiguous discourse segments that are attributed to the same source.</Paragraph>
      <Paragraph position="12"> (c) Group contiguous discourse segments that belong to the same sentence (marked by periods, commas, semicolons, or colons).</Paragraph>
      <Paragraph position="13"> (d) Group contiguous discourse segments that are topically centered on the same entities or events.</Paragraph>
      <Paragraph position="14"> 3. Determine coherence relations between discourse segments and groups of  discourse segments. For each previously unconnected (group of) discourse segment(s), test whether it connects to any of the (groups of) discourse segments that have already been connected to the already existing representation of discourse structure. Use the following steps for each decision:  (a) Use pairs of quotation marks as a signal for attribution.</Paragraph>
      <Paragraph position="15"> (b) For pairs of (groups of) discourse segments that are already connected with one of the contentful coordinating conjunctions from Table 1, choose the coherence relation that corresponds to the coordinating conjunction.</Paragraph>
      <Paragraph position="16"> (c) For pairs of (groups of) discourse segments that are not connected with one of the contentful coordinating conjunctions from  i. Mentally connect the (groups of) discourse segments with one of the coordinating conjunctions from Table 1 and judge whether the resultant passage sounds acceptable.</Paragraph>
      <Paragraph position="17"> ii. If the passage sounds acceptable, choose the coherence relation that corresponds to the coordinating conjunction selected in step (c.i).</Paragraph>
      <Paragraph position="18"> iii. If the passage does not sound acceptable, repeat step (c.i) until an acceptable coordinating conjunction is found.</Paragraph>
      <Paragraph position="19"> iv. If the passage does not sound acceptable with any of the coordinating conjunctions from Table 1, assume that the (groups of) discourse segments under consideration are not related by a coherence relation.</Paragraph>
      <Paragraph position="20"> (d) Iterative procedure for steps (a) and (b): i. Start with any of the unambiguous coordinating conjunctions from Table 1 (because, although, if...then, ... said, for example).</Paragraph>
      <Paragraph position="21">  Wolf and Gibson Representing Discourse Coherence Table 3 Statistics for texts in our database.</Paragraph>
      <Paragraph position="22"> Number of words Number of discourse segments  ii. If none of the unambiguous coordinating conjunctions results in an acceptable passage, use the more ambiguous coordinating conjunctions (and, but, while, also,etc.).</Paragraph>
      <Paragraph position="23"> (e) Important distinctions for steps (2) and (3) (this is based on practical issues that came up during the annotation project): i. Example versus elaboration:Anexample relation sets up an additional entity or event (the example), whereas an elaboration relation provides more details about an already introduced entity or event (the one on which one elaborates). ii. Cause-effect versus temporal sequence:Bothcause-effect and temporal sequence describe a temporal order of events (in cause-effect, the cause has to precede the effect). However, only cause-effect relations have a causal relation between what is stated by the (groups of) discourse segments under consideration. Thus, if there is a causal relation between the (groups of) discourse segments under consideration, assume cause-effect rather than temporal sequence (cf. Lascarides and Asher 1993).</Paragraph>
    </Section>
    <Section position="6" start_page="258" end_page="258" type="sub_section">
      <SectionTitle>
2.5 Annotators
</SectionTitle>
      <Paragraph position="0"> The annotators for the database were MIT undergraduate students who worked in our lab as research students. For training, the annotators received a manual that described the background of the project, discourse segmentation, coherence relations and how to recognize them, and how to use the annotation tools that we developed in our lab (Wolf et al. 2003). The first author of this article provided training for the annotators. Training consisted of explaining the background of the project and the annotation method and of annotating example texts (these texts are not included in our database). Training took 8-10 hours in total, distributed over five days of a week. After completing the training, annotators worked independently.</Paragraph>
    </Section>
    <Section position="7" start_page="258" end_page="467" type="sub_section">
      <SectionTitle>
2.6 Statistics on Annotated Database
</SectionTitle>
      <Paragraph position="0"> In order to evaluate hypotheses about appropriate data structures for representing coherence structures, we have collected a database of 135 texts from the Wall Street Journal 1987-1989 (30 texts) and the AP Newswire 1989 (105 texts) (both from Harman and Liberman [1993]) in which the relations between discourse segments have been labeled with the coherence relations described above. Table 3 shows statistics for this database.</Paragraph>
      <Paragraph position="1">  Computational Linguistics Volume 31, Number 2 Steps 2 (discourse segment grouping) and 3 (coherence relation annotation) of the coding procedure described in section 2.4 were performed independently by two annotators. For step 1 (discourse segmentation), a pilot study on 10 texts showed that agreement on this step, as determined by number of common segments/(number of common segments + number of differing segments), was never below 90%. Therefore, all 135 texts were segmented by two annotators together, resulting in segmentations that both annotators could agree on.</Paragraph>
      <Paragraph position="2"> In order to determine interannotator agreement for step 2 of the coding procedure for the database of annotated texts, we calculated kappa statistics (Carletta 1996). We used the following procedure to construct a confusion matrix: First, all groups marked by either annotator were extracted. Annotator 1 had marked 2,616 groups, and annotator 2 had marked 3,021 groups in the whole database. The groups marked by the annotators consisted of 536 different discourse segment group types (for example, groups that included the first two discourse segments of each text were marked 31 times by both annotators; groups that included the first three discourse segments of each text were marked 6 times by both annotators). Therefore, the confusion matrix had 536 rows and columns. For all annotations of the 135 texts, the agreement was 0.8449, per chance agreement was 0.0161, and kappa was 0.8424. Annotator agreement did not differ as a function of text length, arc length, or kind of coherence relation (all kh  values &lt; 1).</Paragraph>
      <Paragraph position="3"> We also calculated kappa statistics to determine interannotator agreement for step 3 of the coding procedure.</Paragraph>
      <Paragraph position="4">  For all annotations of the 135 texts, the agreement was 0.8761, per chance agreement was 0.2466, and kappa was 0.8355. Annotator agreement did not differ as a function of text length (kh  &lt; 1). Table 4 shows the confusion matrix for the database of 135 annotated texts that was used to compute the kappa statistics. The table shows, for example, that much of the interannotator disagreement seems to have been driven by disagreement over how to annotate elaboration relations (in the whole database, annotator 1 marked 260 elaboration relations where annotator 2 marked no relation; annotator 2 marked 467 elaboration relations where annotator 1 marked no relation). The only other comparable discourse annotation project that we are currently aware of is that of Carlson, Marcu, and Okurowski (2002).</Paragraph>
      <Paragraph position="5">  Since they use trees and split the annotation process into different substeps than those in our procedure, their annotator agreement figures are not directly comparable to ours. Furthermore, note that Carlson and her colleagues do not report annotator agreement figures for their database as a whole, but for different subsets of four to seven documents that were each annotated by different pairs of annotators. For discourse segmentation, they report kappa values ranging from 0.951 to 1.00; for annotation of discourse tree spans, their kappa values ranged from 0.778 to 0.929; for annotation of coherence relation nuclearity (whether a node in a discourse tree is a nucleus or a satellite, cf. section 2.3 for the definition of these terms), kappa values ranged from 0.695 to 0.882; for assigning types of coherence relations, they reported kappa values ranging from 0.624 to 0.823.</Paragraph>
      <Paragraph position="6"> 1 Note that interannotator agreement for step 3 was influenced by interannotator agreement for step 2. For example, one annotator might mark a group of discourse segments 2 and 3, whereas the second annotator might not mark that group of discourse segments. If the first annotator then marks, for example, a cause-effect coherence relation between discourse segment 4 and the group of discourse segments 2 and 3, whereas the second annotator marks a cause-effect coherence relation between discourse segment 4 and discourse segment 3, this would count as a disagreement. Thus, our measure of interannotator agreement for step 3 is conservative.</Paragraph>
      <Paragraph position="7">  In order to represent the coherence relations between discourse segments in a text, most accounts of discourse coherence assume tree structures (Britton 1994; Carlson, Marcu, and Okurowski 2002; Corston-Oliver 1998; Longacre 1983; Grosz and Sidner 1986; Mann and Thompson 1988; Marcu 2000; Polanyi and Scha 1984; Polanyi 1996; Polanyi et al. 2004; van Dijk and Kintsch 1983; Walker 1998); some accounts do not allow crossed dependencies but appear to allow nodes with multiple parents (Lascarides and Asher 1991).</Paragraph>
      <Paragraph position="8">  Other accounts assume less constrained graphs that allow crossed dependencies as well as nodes with multiple parents (e.g., Bergler 1991; Birnbaum 1982; Danlos 2004; Hobbs 1985; McKeown 1985; Reichman 1985; Zukerman and McConachy 1995; for dialogue structure, Penstein Rose et al. 1995).</Paragraph>
      <Paragraph position="9"> Some proponents of tree structures assume that trees are easier to formalize and to derive than less constrained graphs (Marcu 2000; Webber et al. 2003). We demonstrate that in fact many coherence structures in naturally occurring texts cannot be adequately represented by trees. Therefore we argue for less constrained graphs in which nodes represent discourse segments and labeled directed arcs represent the coherence relations that hold between these discourse segments as an appropriate data structure for representing coherence.</Paragraph>
      <Paragraph position="10"> Some proponents of more general graphs argue that trees cannot account for a full discourse structure that represents informational, intentional, and attentional discourse relations. For example, Moore and Pollack (1992) point out that rhetorical structure theory (Mann and Thompson 1988) has both informational and intentional coherence relations but then forces annotators to decide on only one coherence relation between any two discourse segments. Moore and Pollack argue that often there is an informational as well as an intentional coherence relation between two discourse segments, which then presents a problem for RST, since only one of the relations can be annotated. Instead, Moore and Pollack propose allowing more than one coherence relation between two discourse segments, which violates the tree constraint of not having nodes with multiple parents.</Paragraph>
      <Paragraph position="11"> Reichman (1985) argues that tree-based story grammars are not sufficient to account for discourse structure. Instead, she argues that in order to account for the intentional structure of discourse, more general data structures are needed. We argue that the same is true for the informational structure of discourse.</Paragraph>
      <Paragraph position="12"> Moore and Pollack (1992), Moser and Moore (1996), and Reichman (1985) argue that trees are insufficient for representing informational, intentional, and attentional discourse structure. Note, however, that the focus of our work is on informational coherence relations, not on intentional relations. That does not mean that we think that attentional or intentional structure should not be part of a full account of discourse structure. Rather, we would like to argue that whereas the above accounts argue against trees for representing informational, intentional, and attentional discourse structure together, we argue that trees are not even descriptively adequate to describe just informational discourse structure by itself.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>