<?xml version="1.0" standalone="yes"?>
<Paper uid="C00-1031">
  <Title>An Empirical Investigation of the Relation Between Discourse Structure and Co-Reference</Title>
  <Section position="5" start_page="209" end_page="211" type="metho">
    <SectionTitle>
3 The Experiment
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="209" end_page="210" type="sub_section">
      <SectionTitle>
3.1 Materials
</SectionTitle>
      <Paragraph position="0"> We used thirty newspaper texts whose lengths varied widely; the mean o- is 408 words and tile standard deviation/t is 376. Tile texts were annotated manually for co-reference relations of identity (Hirschman and Chinchef, 1997). Tile co-reference relations define equivalence classes oil the set of all marked referents in a text. Tile texts were also manually annotated by Marcu et al. (1999) with disconrse structures built in the style of Mann and Thompson (1988). Each discourse analysis yielded an average of 52 elementary discourse units.</Paragraph>
      <Paragraph position="1"> See (Hirschman and Chinchor, 1997) and (Marcu et al., 1999) for details of tile annotation processes.</Paragraph>
      <Paragraph position="3"/>
    </Section>
    <Section position="2" start_page="210" end_page="211" type="sub_section">
      <SectionTitle>
3.2 Comparing potential to establish co-referential
links
3.2.1 Method
</SectionTitle>
      <Paragraph position="0"> The annotations for co-reference rchttions and rhetorical struclure trues for the thirty texts were fused, yielding representations that rcllect not only tile discourse strutlure, but also the co-reference equivalencc classes specitic to each tcxl. Based on this information, we cvalualed the potential of each of the two classes (51&amp;quot; models discussed in secdon 2 (Linear-k and Discourse-VT-k) to correctly establish co-referential links as follows: For each model, each k, and each marked referential expression o., we determined whether or not tlle corresponding LPA (delined over k elementary units) contained a referee from the same equiwdence class. For example, for the Linear-2 model and referential expression lhe .vmaller company in t, nit 9, we estimated whether a co-refercntial link could be established between the smaller company and another referential expression in units 7, 8, or 9.</Paragraph>
      <Paragraph position="1"> For the Discourse-VT-2 model and the same referential expression, we estimated whether a co-referential link could bE established between the smaller company and another referential expression in units 1, 8, or 9, which correspond to the DRA of unit 9.</Paragraph>
      <Paragraph position="2"> qb enable a fair comparison of the two models, when k is la,'ger than the size of the DRA of a given unit, WE extend thatDRA using the closest units that precede the unit under scrutiny and are not ah'eady in the DRA. Hence, for the Linear-3 model and the referential expression the smaller conq~any in trait 9, we estimate whether a co-referential link can be established between the xmaller company and another referential expression in units 9, 8, 7, or 6. For tile Discourse-VT-3 model and tile same rcfermltial expression, we estimate whclher a co-referential link can be established between the smaller company and another referential expression in units 9, 8, 1, or 7, which correspond I(5 the DRA of mill 9 (unfls 9, 8, and 1) and to unit 7, the closest unit preceding unit 9 that is not ill ils I)RA.</Paragraph>
      <Paragraph position="3"> For the l)iscottrse-VT-k models, we assume Ihat the Fxtended DRA (EDRA) of size \]c of a unit ~t. (EDRAz: ('~)) is given by the lh'st 1 _&lt; k units of a sequence that lists, in reverse order, the units of the DRA of '~z plus the /c - l units that precede tt but arc not in its DRA.</Paragraph>
      <Paragraph position="4"> For example, \['or the text in Figure 1, the following relations hold: EDRAc,(!)) = 9; EDRAI(C.)) =</Paragraph>
      <Paragraph position="6"> EDRAz:(u) is given by u and the k units that immediately precede ~t.</Paragraph>
      <Paragraph position="7"> The potential p(M, a, EDRA,~) (5t' a model M to determine correct co-referential links with respect to a ref-Erential expression a in unit u, given a corresponding EDRA of size k (EDRAt.(u)), is assigned the value 1 if the EDRA contains a co-referent from the same equiwtlence class as a. Otherwise, p(M, ,, EDRAt~) is assigned the value O. The potential p(k4, 6', k) of a model M to determine correct co-rEferential links for all referential expressions in a corpus of texts C, using EDRAs of size k, is computed as the SUlll oF the potentials p(M, a.,EI)RA#) of all referential expressions ct in C'.</Paragraph>
      <Paragraph position="8"> This potential is normalized to a wdue bEtweEn 0 and  expressions in the corpus that have an antecedent.</Paragraph>
      <Paragraph position="9"> By examining the potential of each model to correctly determine co-referential expressions for each k, it is possible to determine the degree to which an implementation of a given approach can contribute to the overall efficiency of anaphora resolution systems. That is, if a given model has the potential to correctly determine a significant percentage of co-referential expressions with small DRAs, an anaphora resolution system implementing that model will have to consider fewer options overall. Hence, the probability of error is reduced.</Paragraph>
    </Section>
  </Section>
  <Section position="6" start_page="211" end_page="212" type="metho">
    <SectionTitle>
3.2.2 Results
</SectionTitle>
    <Paragraph position="0"> The graph in Figure 3 shows the potentials of the Linear-k and Discourse-VT-k models to correctly determine co-referential links for each k from 1 to 20. The graph in Figure 4 represents the same potefftials but focuses only on ks in the interval \[2,9\]. As these two graphs show, the potentials increase monotonically with k, the VT-k models always doing better than the Linear-k models. Eventually, for large ks, the potential performance of the two models converges to 100%.</Paragraph>
    <Paragraph position="1"> The graphs in Figures 3 and 4 also suggest resolution strategies for implemented systems. For example, the graphs suggests that by choosing to work with EDRAs of size 7, a discourse-based system has the potential of resolving more than 90% of the co-referential links in a text correctly. To achieve the same potential, a linear-based system needs to look back 8 units. Ifa system does not look back at all and attempts to resolve co-referential links only within the unit under scrutiny (k = 0), it has the potential to correctly resolve about 40% of the co-referential links.</Paragraph>
    <Paragraph position="2"> To provide a clearer idea of how the two models differ, Figure 5 shows, for each k, the value of the Discourse-VT-k potentials divided by the value of the Linear-k potentials. For k = 0, the potentials of both models are equal because both use only the unit in focus in order to determine co-referential links. For k = 1, the Discourse-VT-1 model is about 7% better than the Linear-! model. As the value of k increases, the value Discourse-VTk/Linear-k converges to 1.</Paragraph>
    <Paragraph position="3"> In Figures 6 and 7, we display the number of exceptions, i.e., co-referential links that Discourse-VT-k and Linear-k models cannot determine correctly. As one can see, over the whole corpus, for each k _&lt; 3, the Discourse-VT-k models have the potential to determine correctly about 100 more co-referential links than the Linear-k models. As k increases, the performance of the two models converges.</Paragraph>
    <Paragraph position="4"> 3,2,3 Statistical significance In order to assess the statistical significance of the difference between the potentials of the two models to establish correct co-referential links, we carried out a Paired-Samples T Test for each k. In general, a Paired-Samples T Test checks whether the mean of casewise differences between two variables differs from 0. For each text in  k models to determine correct co-referential links (2 &lt; k _&lt; 9).</Paragraph>
    <Paragraph position="5"> the corpus and each k, we determined the potentials of both VT-k and Linear-k models to establish correct co-referential links in that text. For ks smaller than 4, the difference in potentials was statistically significant. For example, for k = 3, t = 3.345, df = 29, P = 0.002. For values of k larger than or equal to 4, the difference was no longer significant. These results are consistent with the graphs shown in Figure 3 to 7, which all show that the potentials of Discourse-VT-k and Linear-k models converges to the same value as the value of k increases.</Paragraph>
    <Section position="1" start_page="211" end_page="212" type="sub_section">
      <SectionTitle>
3.3 Comparing the effort required to establish
co-referential links
3.3.1 Method
</SectionTitle>
      <Paragraph position="0"> The method described in section 3.2.1 estimates the potential of Linear-k and Discourse-VT-k models to determine correct co-referential links by treating EDRAs as sets. However, from a computational perspective (and  be correctly determined by Discourse-VT-k and Linear-k models (0 &lt; /~' &lt; 20).</Paragraph>
      <Paragraph position="1"> presumably, from a psycholinguistic perspective as well) it also makes sense to compare the effort required by the two classes of models to establish correct co-referential links. We estimate this effort using a very simple metric that assumes that the closer an antecedent is to a corresponding referential expression in the EDRA, the better. Hence, in estimating the effort to establish a co-referential link, we treat EDRAs as ordered lists. For example, using the Linear-9 model, to determine the correct antecedent of the referential expression the smaller company in unit 9 of Figure 1, it is necessary to search back through 4 units (to unit 5, which contains the referent Genetic Therapy). Had unit 5 been Ml: Cassey succeeds M.</Paragraph>
      <Paragraph position="2"> James Barrett, 50, we would have had to go back 8 units (to unit 1) in order to correctly resolve the RE the smaller company. In contrast, in the Discourse-VT-9 model, we go back only 2 units because unit 1 is two units away fi'om unit 9 (EDRA:~ (9) = 9, 8, 1,7, 6, 5,4, 3, 2).</Paragraph>
      <Paragraph position="3"> We consider that the effort e(AJ, a, EDRAa.) of a model M to determine correct co-referential links with respect to one referential, in unit u, given a corresponding EDRA of size L&amp;quot; (EDRA~.(,)) is given by the number of units between u and the first unit in EDRAk(u) that contains a co.-referential expression of a.</Paragraph>
      <Paragraph position="4"> The effort e(M, C, k) of a model M to determine correct co-referential links for all referential expressions in a corpus of texts C using EDRAs of size k was computed as the sum of the efforts e(M, a, EDRAk) of all referentia ! expressions a in C.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>