<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-2703">
  <Title>Annotating Discourse Connectives And Their Arguments</Title>
  <Section position="3" start_page="0" end_page="0" type="metho">
    <SectionTitle>
2 Theoretical background
</SectionTitle>
    <Paragraph position="0"> The annotation project presented in this paper builds on basic ideas presented in Webber and Joshi (1998), Webber et al. (1999b) and Webber et al. (2003) - that connectives are discourse-level predicates which project predicate-argument structure on a par with verbs at the sentence level. Webber and Joshi (1998) propose a tree-adjoining grammar for discourse (DLTAG) in which compositional aspects of discourse meaning are formally defined, thus teasing apart compositional from non-compositional layers of meaning. In this framework, connectives are grouped into natural classes depending on the structure that they project at the discourse level. Subordinate and coordinating conjunctions, for example, require two arguments that can be identified structurally from adjacent units of discourse. What Webber et al. (2003) call anaphoric discourse connectives (some, but not all, discourse adverbials, such as &amp;quot;otherwise&amp;quot;, &amp;quot;instead&amp;quot;, &amp;quot;furthermore&amp;quot;, etc.) also require two arguments, but only one of them derives structurally. For the complete interpretation of these connectives, their other argument needs to be recovered. The crucial contribution of this framework to the design of the current project is what can be seen as a bottom-up approach to discourse structure. Specifically, instead of appealing to an abstract (and arbitrary) set of discourse relations whose identification involves confounding multiple sources of discourse meaning, we start with the annotation of discourse connectives and their arguments, thus exposing a clearly defined level of discourse representation.</Paragraph>
  </Section>
  <Section position="4" start_page="0" end_page="0" type="metho">
    <SectionTitle>
3 Project description
</SectionTitle>
    <Paragraph position="0"> The PTDB project began in November 2002. The first phase, including pilot annotations and preliminary development of guidelines, was completed in May 2003. The PDTB is expected to be released by November 2005. Intermediate versions of the annotated corpus will be made available for receiving feedback.</Paragraph>
    <Paragraph position="1"> The PDTB corpus will include annotations of four types of connectives: subordinating conjunctions, coordinating conjunctions, adverbial connectives and implicit connectives. We specify each of these types in more detail in Section 3.1. The final number of annotations in the corpus will amount to approximately 30,000; 10,000 implicit connectives and 20,000 annotations of the 250 explicit connectives identified in the corpus. The final version of the corpus will also contain characterizations of the semantic roles associated with the arguments of each type of connective.</Paragraph>
    <Paragraph position="2"> In this paper we present the results of annotating 10 explicit connectives, amounting to a total of 2717 annotations, as well as 386 tokens of implicit connectives. The set of 10 connectives comprises the adverbial connectives 'therefore', 'as a result', 'instead', 'otherwise', 'nevertheless', and the subordinate conjunctions 'because', 'although', 'even though', 'when', and 'so that'. In all cases, annotations have been performed by four annotators. While this slows down the annotation process considerably, the nature, significance and magnitude of the project as well as the well-known complexity of discourse annotation tasks impels us to strive for maximum reliability, achieved by having the task performed by multiple annotators.3 Individual annotation proceeds one connective at a time. The annotation tool WordFreak4 is used to identify all instances of the given connective in the corpus, and these are then annotated independently and manually by four annotators. This way, the annotators quickly gain experience with that connective and develop a better understanding of its predicate-argument characteristics.</Paragraph>
    <Paragraph position="3"> Similarly, for the annotation of implicit connectives, all instances (as specified in the guidelines, see Section 3.2) are identified one file at a time. For this task, the annotators are required to read the entire file so that they can make well-informed and reliable decisions about the implicit connectives and their arguments. In addition, after the arguments of each implicit connective have been identified, the annotators provide, if possible, an explicit connective (or other suitable expression) that best expresses the inferred relation. As with explicit connectives, annotations of implicit connectives are done by four annota3When inter-annotator consistency has stabilized, we intend to reduce the number of annotators to three, or maybe two at the minimum.</Paragraph>
    <Paragraph position="4"> 4WordFreak was developed by Tom Morton at the University of Pennsylvania. It has been substantially modified by Jeremy Lacivita to fit the needs of the PDTB project. A snapshot of the tool can be seen at http://www.cis.upenn.edu/ pdtb.</Paragraph>
    <Paragraph position="5"> tors.</Paragraph>
    <Paragraph position="6"> Compared with Propbank's annotation of verb predicate-argument structures, annotation of arguments of discourse predicates is different in interesting ways.</Paragraph>
    <Paragraph position="7"> Propbank annotators have to determine the number of arguments required by each verb. In contrast, discourse connectives exhibit a clear predicate-argument structure requiring only two arguments. The main challenge we have discovered for annotating discourse connectives is determining the extent of their arguments. Even subordinate conjunctions whose arguments never cross a sentence boundary may sometimes be the source of disagreement between annotators.</Paragraph>
    <Paragraph position="8"> In what follows, we present a brief overview of the classes of connectives that we annotate, followed by highlights of the annotation manual and relevant corpus examples.</Paragraph>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
3.1 Discourse connectives
</SectionTitle>
      <Paragraph position="0"> We classify discourse connectives into four classes: subordinate and coordinating conjunctions, adverbials and implicit connectives. Examples of each type are given below, with their arguments shown in square brackets and the connectives, in italics.</Paragraph>
      <Paragraph position="1">  Subordinate conjunctions introduce clauses that are syntactically dependent on a main clause. The most common types of relations that they express are temporal (e.g., 'when', 'as soon as'), causal e.g., 'because'), concessive (e.g., 'although', 'even though'), purpose (e.g., 'so that', 'in order that') and conditional (e.g., 'if', 'unless). Clauses introduced with a subordinate conjunction may be preposed (or, more rarely, interposed) with respect to the main clause, as shown in (1).</Paragraph>
      <Paragraph position="2">  (1) Because [the drought reduced U.S. stockpiles], [they have more than enough storage space for their new crop], and that permits them to wait for prices to rise.  Coordinating conjunctions are ones such as 'and', 'but', and 'or'. Example (2) shows the annotation of an instance of the conjunction 'and'.</Paragraph>
      <Paragraph position="3"> (2) [William Gates and Paul Allen in 1975 developed an early language-housekeeper system for PCs], and [Gates became an industry billionaire six years after IBM adapted one of these versions in 1981].</Paragraph>
      <Paragraph position="4"> Instances of coordinating conjunctions which coordinate nominal or other non-clausal constituents are excluded from annotation. We also exclude cases of VPcoordination because in such cases the arguments of the connective can be retrieved automatically from the syntactic layer.</Paragraph>
      <Paragraph position="5">  Adverbial connectives are sentence-modifying adverbs which express a discourse relation (Forbes, 2003). The class of adverbial connectives includes 'however', 'therefore', 'then', 'otherwise', etc. In this class, we have also included prepositional phrases with a similar sentence modifying function such as 'as a result', 'in addition', 'in fact', etc. Example (3) shows the annotation of an instance of the adverbial connective 'as a result'.</Paragraph>
      <Paragraph position="6"> (3) ...[many analysts expected energy prices to rise at the consumer level too]. As a result, [many economists were expecting the consumer price index to increase significantly more than it did].</Paragraph>
      <Paragraph position="7"> The arguments of adverbial connectives may or may not be adjacent to the sentence containing the connective. In a few cases, an argument may be found one or two paragraphs away from the connective.</Paragraph>
      <Paragraph position="8">  Implicit connectives are identified between adjacent sentences with no explicit connectives.5 The annotation of implicit connectives is intended to capture the connection between two sentences appearing in adjacent positions. For example, in (4), the two adjacent sentences are connected in a way similar to having the explicit connective &amp;quot;but&amp;quot; contrasting them. Indeed, for implicit connectives, annotators are asked to provide, when possible, an explicit connective that best describes the inferred relation. The explicit connective provided in (4) was 'in contrast'.</Paragraph>
      <Paragraph position="9">  (4) ...[The $6 billion that some 40 companies are looking to raise in the year ending March 31 compares with only $2.7 billion raised on the capital market in the previous fiscal year]. IMPLICIT-(In contrast) [In fiscal 1984 before Mr. Gandhi came to power, only $810 million was raised].</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
3.2 Annotation guidelines
</SectionTitle>
      <Paragraph position="0"> The annotation guidelines for PDTB have been revised considerably since the pilot phase of the project in May 2003. The current version of the guidelines is available at http://www.cis.upenn.edu/ pdtb. Below we outline the basic points.</Paragraph>
      <Paragraph position="1"> 3.2.1 What counts as a discourse connective? We count as discourse connectives (1) all subordinating and coordinating conjunctions, (2) certain adverbials, and (3) implicit connectives. The adverbials include only those which convey a relation between events or states.</Paragraph>
      <Paragraph position="2"> For example, in (5) 'as a result' conveys a cause-effect relation between the event of limiting the size of new steel 5There are, of course, other implicit connectives that we are not taking into account.</Paragraph>
      <Paragraph position="3"> mills and that of the industry operating out of small, expensive and highly inefficient units. In contrast, the semantic interpretation of 'strangely' in (6) only requires a single event/state which it classifies in the set of strange  (5) [In the past, the socialist policies of the government strictly limited the size of new steel mills, petrochemical plants, car factories and other industrial concerns to conserve resources and restrict the profits businessmen could make]. As a result, industry operated out of small, expensive, highly inefficient industrial units.</Paragraph>
      <Paragraph position="4"> (6) Strangely, conventional wisdom inside the Beltway re- null gards these transfer payments as &amp;quot;uncontrollable&amp;quot; or &amp;quot;nondiscretionary.&amp;quot; The guidelines also highlight instances of lexical items with multiple functions, only one of which is as a discourse connective. For example, 'when' can either serve as a subordinate conjunction or introduce a relative clause modifying a nominal phrase, as in (7), where the when-clause modifies the nominal '1985'.7Here we again benefit from building discourse annotation on top of Penn TreeBank because the syntactic annotation of when-clauses distinguishes the two functions: When-relatives are marked as NP-modifiers adjoining to an NP, whereas adverbial when-clauses adjoin to a sentential node.</Paragraph>
      <Paragraph position="5"> (7) Attorneys have argued since 1985, when the law took effect.</Paragraph>
      <Paragraph position="6"> Similarly, some since-clauses function as NP modifiers as shown in (8). In such cases, 'since' is not annotated as a connective. As in the case of when-clauses, instances of NP modifying since-clauses can be identified in the Penn TreeBank by virtue of their syntactic annotation.</Paragraph>
      <Paragraph position="7"> (8) In the decade since the communist nation emerged from isolation, its burgeoning trade with the West has lifted Hong Kong's status as a regional business partner.</Paragraph>
      <Paragraph position="8"> Finally, implicit connectives count as connectives.</Paragraph>
      <Paragraph position="9"> They are identified between adjacent sentences which do not contain any other explicit connectives. Currently, we are not annotating implicit connectives intra-sententially, such as between the matrix clause and free adjunct in Example (9). We plan to incorporate annotations of implicit intra-sentential connectives at a later stage of the project.  (9) Second, they channel monthly mortgage payments into semiannual payments, reducing the administrative burden on investors.</Paragraph>
      <Paragraph position="10">  Because we take discourse relations to hold between abstract objects, we require that an argument contains at least one predicate along with its arguments. Of course, a sequence of clauses or sentences may also form a legal argument, containing multiple predicates.</Paragraph>
      <Paragraph position="11"> Because our annotations are done directly on top of the Penn TreeBank, annotators may select as an argument certain textual spans that appear to exclude one or more arguments of the predicate. These are cases in which these arguments are directly retrievable from the syntactic annotation. Thus, we are able to select only the predicates that are required for the interpretation of the discourse connective and simultaneously access their arguments for the complete interpretation of the clause while keeping the annotations of single arguments simple and maximally contiguous. In (10), for example, the relative clause is marked as one of the two arguments of the connective 'even though'. The subject of the verb in the relative clause is directly retrievable from the Penn TreeBank annotation. Similarly, in (11) the subject of the infinitival clause is also available from the syntactic representation. (10) Workers described &amp;quot;clouds of dust&amp;quot; [that hung over parts of the factory] even though [exhaust fans ventilated the air].</Paragraph>
      <Paragraph position="12"> (11) The average maturity for funds open only to institutions, considered by some [to be a stronger indicator] because [those managers watch the market closely], reached a high point for the year - 33 days.</Paragraph>
      <Paragraph position="13"> There are two exceptions to the requirement that an argument include a verb - these are nominal phrases that express an event or a state, and discourse deictics that denote an event or state. In (12), for example, the nominal phrase 'fainting spells' can be marked as a legal argument of the connective 'when' because the phrase expresses an event of fainting.</Paragraph>
      <Paragraph position="14"> (12) Its symptoms include a cold sweat at the sound of debate, clammy hands in the face of congressional criticism, and [fainting spells] when [someone writes the word &amp;quot;controversy.&amp;quot;] Discourse deictic expressions are forms such as 'this' and 'that' that can be used to denote the interpretation of clausal textual spans from the preceding discourse.</Paragraph>
      <Paragraph position="15"> In (13), for example, 'that' denotes the interpretation of the sentence immediately preceding it. Our annotators are guided to make argument selections that assume that anaphoric and deictic expressions have been resolved.</Paragraph>
      <Paragraph position="16"> Thus, in (13), they are able to select 'That's' as one argument of the connective 'because'.</Paragraph>
      <Paragraph position="17"> (13) Airline stocks typically sell at a discount of about one-third to the stock market's price-earnings ratio - which is currently about 13 times earnings. [That's] because [airline earnings, like those of auto makers, have been subject to the cyclical ups-and-downs of the economy].</Paragraph>
      <Paragraph position="18"> The annotators are also informed that in some cases, an argument of a connective must be derived from the selected textual span (Webber et al., 1999a; Webber et al., 2003). This is the case for the first argument of 'instead' in (14), which does not include the negation, although it is contained in the selected text.8 (14) [No price for the new shares has been set]. Instead, [the companies will leave it up to the marketplace to decide]. In sum, legal arguments can be groups of sentences, single sentences (a main clause and its subordinate clauses), single clauses (tensed or non-tensed), NPs that specify events or situations, and discourse deictic expressions. null 3.2.3 How far does an argument extend? One particularly significant addition to the guidelines came as a result of differences among annotators as to how large a span constituted the argument of a connective. During pilot annotations, annotators used three annotation tags: CONN for the connective and ARG1 and ARG2 for the two arguments. To this set, we have added the optional tags SUP1, SUP2 (supplementary) for cases when the annotator wants to mark textual spans s/he considers to be useful, supplementary information for the interpretation an argument. Example (15) demonstrates the use of SUP1. Arguments are shown in square brackets, while spans providing supplementary information are shown in parentheses.</Paragraph>
      <Paragraph position="19"> (15) Although [started in 1965], [Wedtech didn't really get rolling until 1975] (when Mr. Neuberger discovered the Federal Government's Section 8 minority business program). null</Paragraph>
    </Section>
  </Section>
  <Section position="5" start_page="0" end_page="0" type="metho">
    <SectionTitle>
4 Data analysis
</SectionTitle>
    <Paragraph position="0"> To test the reliability of the annotation, we first considered the kappa statistic (Siegel and Castellan, 1988) which is used extensively in empirical studies of discourse (Carletta, 1996). The kappa coefficient provides an inter-annotator agreement figure for any number of annotators by measuring pairwise agreement between them and by correcting for chance expected agreement. However, the statistic requires the data tokens to be classified into discrete categories, and as a result, we could not apply it to our data since the PDTB annotation tokens cannot be classified as such. Rather, annotation in the PDTB constitutes either selection of a span of text for the arguments of connectives which can be of indeterminate length or providing explicit expressions for implicit connectives from an open-ended class of expressions.</Paragraph>
    <Paragraph position="1"> 8For a preliminary corpus-based analysis of the arguments of 'instead', see Miltsakaki et al. (2003).</Paragraph>
    <Paragraph position="2"> Instead, we have assessed inter-annotator agreement in terms of agreement/disagreement on span or named expression identity for each token as a percentage of the pairs of spans or expressions that actually matched versus those that should have. For the argument annotations, we use a most conservative measure - the exact match criterion. In addition, we also used different diagnostics for the argument annotations for the explicit connectives, reporting percentage agreement on different classes of tokens, such as those in which the first argument (ARG1) annotations and second argument (ARG2) annotations were counted independently, as well as those in which the ARG1 and ARG2 annotations (for each connective) were counted together as a single token. For all the argument annotations, the computation of agreement excluded the supplementary annotations (cf. Section 3.2.3).</Paragraph>
    <Paragraph position="3"> We present here agreement results on ARG1 and ARG2 annotations by two annotators for the annotation of ten explicit connectives, amounting to a total of 2717 annotations, and 368 annotations of implicit connectives, including agreement results on the explicit expression the annotators used in in place of the implicit connectives as well as the ARG1 and ARG2 annotations of the implicit connectives.9 The ten explicit connectives include 5 subordinating conjunctions (when, because, even though, although, and so that) and 5 adverbials (nevertheless, otherwise, instead, therefore, and as a result).</Paragraph>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
4.1 Inter-annotator Agreement
</SectionTitle>
      <Paragraph position="0"> For the explicit connective annotations, we used two diagnostics for measuring inter-annotator agreement. In the first diagnostic , we took the class of tokens as the total number of argument annotations, treating ARG1 and ARG2 annotations as independent tokens. The total number of tokens in this class is therefore twice the number of connective tokens, i.e, 5434. We recorded agreement using the exact match criterion. That is, for any ARG1 or ARG2 token, agreement was recorded as 1 when both annotators made identical textual selections for the annotation and 0 when the annotators made non-identical selections.</Paragraph>
      <Paragraph position="1"> We achieved 90.2% agreement (4900/5434 tokens) on the annotations for this class. Agreement on only ARG1 tokens was 86.3%, and agreement on only ARG2 tokens was 94.1%. Further distribution of the agreements by connective is given in Table 1. Connectives are grouped in the table by type (subordinating conjunction (SUBCONJ) and adverbial (ADV)). The second col9Right now SUP1 and SUP2 annotations are for our use only and are not included in the current evaluations. Additional annotations by another 2 annotators are currently underway. The 2 annotators of the explicit connectives are different from the 2 annotators of the implicit connectives.</Paragraph>
      <Paragraph position="2"> umn gives the number of agreeing tokens for each connective and the third column gives the total number of (ARG1+ARG2) tokens available for that connective. The last column gives the percent agreement for the connective in that row, i.e., as a percentage of tokens for which agreement was 1 (column 2) versus the total number of tokens for that connective (column 3).</Paragraph>
      <Paragraph position="3">  The table shows that we achieved high agreement on argument annotations of subordinating conjunctions (92.4%). Average agreement on the adverbials was lower (71.8%). This difference between the two types is not surprising, since locating the anaphoric (ARG1) argument of adverbial connectives is believed to be a harder task than that of locating the arguments of subordinating conjunctions. For example, the anaphoric argument of the adverbial connectives may be located in some non-adjacent span of text, even several paragraphs away. Arguments of subordinating conjunctions, on the other hand, can most often be found in spans of text adjacent to the connective. The table also shows that there was uniform agreement across the different subordinating conjunctions (roughly 90%), whereas the adverbials showed more variation.</Paragraph>
      <Paragraph position="4"> In particular, agreement on otherwise and therefore was high (95.7% and 87.5% respectively), while lower for the other three adverbials, instead (72.9%), as a result (65.5%), and nevertheless (59.6%). This suggests either greater variability in how these adverbials are interpreted or greater complexity in their interpretation, which results in more variability when people are forced to associate an interpretation with a particular text span.</Paragraph>
      <Paragraph position="5"> We also computed agreement using a second more conservative diagnostic in which we took the class of tokens as the total number of connective tokens (2717) so that the ARG1 and ARG2 annotations for each connective were treated together as part of the same token. Here again, we recorded agreement using the exact match measure. That is, for any connective token, agreement was recorded as 1 when both annotators made identical textual selections for the annotation of both arguments and 0 when the annotators made non-identical selections for any one or both arguments.</Paragraph>
      <Paragraph position="6"> We achieved 82.8% agreement (2249/2717 tokens) on the annotations for this class. Table 2 gives the distribution of the agreements by connective. The table shows relatively lower agreements when compared with the first diagnostic, for both subordinating conjunctions (86%) as well as adverbials (57%). However, this difference is understandable since the token class as defined for this diagnostic yields a stricter measure of agreement.</Paragraph>
      <Paragraph position="7">  We classified disagreements into 4 major types. The result of classifying the 534 disagreements from Diagnostic 1 (Table 1) is given in Table 3. The third column gives the percent of the total disagreements for each type.  The majority of disagreements (79%) were due to Partial Overlap, which subsumes the categories Higher Verb, Dependent Clause, Parenthetical and Other. Partial Overlap means that there was partial overlap in the annotations selected by the two annotators. Higher verb includes tokens where one of the annotators included the governing predicate for the clause marked by both annotators. The higher clause occurred on the left or right periphery of the lower clause. Dependent Clause includes tokens where one of the annotators included extra clausal material that is syntactically dependent on the clause that was selected by both, and that occurs on the left or right periphery of the common text. Parenthetical means that one of the annotators included a medial parenthetical, while the other did not. The intervening text could be the main as well as the dependent clause. An example is provided below: (16) Bankers said [warrants for Hong Kong stocks are attractive] because [they give foreign investors], wary of volatility in the colony's stock market, [an opportunity to buy shares without taking too great a risk].</Paragraph>
      <Paragraph position="8"> (17) Bankers said [warrants for Hong Kong stocks are attractive] because [they give foreign investors, wary of volatility in the colony's stock market, an opportunity to buy shares without taking too great a risk].</Paragraph>
      <Paragraph position="9"> Other included tokens with partial overlap between annotations, but in addition included a combination of more than type, such as higher verb+dependent clause.</Paragraph>
      <Paragraph position="10"> Note that disagreements that contain a partial overlap could be counted as agreeing tokens if we relaxed the more conservative exact match measure to a partial match measure. Our subjective view was that in several cases, the &amp;quot;extra&amp;quot; textual material, especially those fitting the dependent clause and parenthetical category did not make any significant semantic contribution in terms of their inclusion or exclusion in the argument. With the partial match measure, excluding these cases reduces the disagreements to half the given number, giving us 94.5% agreement overall.</Paragraph>
      <Paragraph position="11"> The No Overlap tokens were cases of true disagreement in that there was no overlap in the annotations selected by the annotators. These tokens constituted 5.6% of the disagreements. Examples (18) and (19) shows the two annotations for a token in which there was no overlap in the ARG1 annotation. Missing Annotations also constituted a substantial proportion of the disagreements (13.5%) and was used for tokens where the annotation was missing for one annotator. Note that these don't really count as disagreement, since all connectives are pretheoretically assumed to require two arguments. Unresolved includes tokens which have introduced new issues for the annotation guidelines and cannot be resolved at this time. These include issues such as how to treat comparatives, certain types of adjunct clauses, certain types of nominalizations etc.</Paragraph>
      <Paragraph position="12"> (18) [The word &amp;quot;death&amp;quot; cannot be escaped entirely by the industry], but salesmen dodge it wherever possible or cloak it in euphemisms, [preferring to talk about &amp;quot;savings&amp;quot; and &amp;quot;investment&amp;quot;] instead.</Paragraph>
      <Paragraph position="13"> (19) The word &amp;quot;death&amp;quot; cannot be escaped entirely by the industry, but salesmen dodge it wherever possible or [cloak it in euphemisms], preferring [to talk about &amp;quot;savings&amp;quot; and &amp;quot;investment&amp;quot;] instead.</Paragraph>
      <Paragraph position="14">  For the 386 tokens of implicit connectives, we analyzed inter-annotator agreement between two annotators for (a) the explicit connectives they provided in place of an implicit connective, and (b) the argument annotations of the implicit connectives.</Paragraph>
      <Paragraph position="15"> As a preliminary step in analyzing agreement on the type of explicit connective provided by the annotators in place of an implicit connective, we considered 5 groups of connectives conveying : a) additional information (e.g., 'furthermore', 'in addition') b) cause-effect relations (e.g., 'because', 'as a result'), c) temporal relations (e.g., 'then', 'simultaneously'), d) contrastive relations (e.g., 'however', 'although'), and e) restatement or summarization (e.g., 'in other words', 'in sum'). 10 Agreement was then computed on these basic groups of connectives.11 From the total of 386 tokens of implicit connectives, 9 were excluded from the analysis due to technical error (missing annotation). For the remaining 307 tokens, we achieved 72% agreement on the type of explicit connective that best conveyed the interpretation of the implicit connective.</Paragraph>
      <Paragraph position="16"> For the argument annotations of the implicit connectives, we present agreement results from using the first diagnostic used for the explicit connectives. That is, we counted ARG1 and ARG2 annotations as independent tokens and computed percent agreement using the exact match criterion. On the 772 ARG1 and ARG2 tokens, we achieved 85.1% (657/772) agreement between 2 annotators. The analysis of the 115 disagreements is given in Table 4. Note that here again, the number of disagreements reduces to half using the partial match measure for the parenthetical and dependent clause classes, giving us 92.6% agreement overall.</Paragraph>
      <Paragraph position="17">  10These groups are based on types of coherence relations derived from corpus-based distributions of connectives presented in (Knott, 1996). Initially, we also considered a group of connectives expressing hypothetical relations but no such connectives were identified in the annotations.</Paragraph>
      <Paragraph position="18"> 11Some polysemous connectives such as 'while' and 'in fact' appeared in more than one group.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>