<?xml version="1.0" standalone="yes"?>
<Paper uid="W96-0412">
  <Title>An Evaluation of Anaphor Generation in Chinese</Title>
  <Section position="5" start_page="112" end_page="113" type="metho">
    <SectionTitle>
3 Systems to Compare and the Test
Task
</SectionTitle>
    <Paragraph position="0"> Having described the framework of the evaluation, in this section, we give details about the object systems to be compared in the evaluation work and the tasks to be performed in the evaluation work.</Paragraph>
    <Section position="1" start_page="112" end_page="113" type="sub_section">
      <SectionTitle>
3.1 Systems to compare
</SectionTitle>
      <Paragraph position="0"> The anaphor generation rule we obtained in our previous studies (YM94; YM95; Yeh95) is shown in Fig. 2, where the internal nodes represent constraints and the terminal nodes are the decisions of using a zero (Z), pronominal (P), or nominal (N) form. The locality constraint checks whether the anaphor in question occurs either in the immediately previous utterance or at a long distance. The second constraint determines whether an anaphor occurs in a position violating syntactic constraints on zero anaphors. We adopted the concept of discourse segment structure in (GS86) to build up the constraint at segment beginning. It checks whether an anaphor is at the beginning of a discourse segment. The salience constraint says that both the positions of an anaphor and its antecedent are the topics of their respective utterances.</Paragraph>
      <Paragraph position="1"> The animacy constraint checks whether the anaphor in question is animate. Then the following rule is used if a nominal form is decided on.</Paragraph>
      <Paragraph position="2"> If a nominal anaphor, n, is at the beginning of a &amp;quot;sentence&amp;quot; 2, or is the first mention of the referent in a &amp;quot;sentence,&amp;quot; then a full description is preferred; otherwise, if n is within a &amp;quot;sentence&amp;quot; or has been mentioned previously in the same &amp;quot;sentence&amp;quot; without distracting elements, then a reduced description is preferred; otherwise a full description is preferred.</Paragraph>
      <Paragraph position="3"> The constraints in the anaphor generation rule were established by consulting relevant linguistic studies (YM94; YM95; Yeh95). Consequently, subsets of constraints in the above rule can be thought of as possible rules, if not complete, for the generation of anaphors in Chinese. As described previously, the systems to compare in this evaluation work are assumed to share the same individual components, except the anaphor generation rules. In this paper, we equipped each system with such a possible anaphor generation rule.</Paragraph>
      <Paragraph position="4"> We chose three rules, termed TR1 TR2 and TR3, with different complexities among the possible candidates as the targets of the test 3. The  stop of a &amp;quot;sentence&amp;quot;; a comma within a &amp;quot;sentence&amp;quot; indicates a temporary stop.</Paragraph>
      <Paragraph position="5"> aThe use of these rules enables us to investigate the effectiveness of individual constraints.</Paragraph>
      <Paragraph position="6">  rules are shown in Fig. 3. The first one uses locality, syntactic constraints and animacy. The second and the third rules have one additional constraint, namely, discourse segment boundaries and salience, respectively, added to their predecessors. In the following, we use the above rule names to represent the systems.</Paragraph>
    </Section>
    <Section position="2" start_page="113" end_page="113" type="sub_section">
      <SectionTitle>
3.2 The test task
</SectionTitle>
      <Paragraph position="0"> The task can be divided into an annotation and a comparison stage. Each of twelve native speakers of Chinese was given a number of test sheets to finish. On each sheet is a text generated by our generation system. Each anaphor position in a generated text was left empty and all candidate forms of the anaphor, including zero, pronominal, and full, or reduced descriptions were put under the empty space. The task for a speaker to perform was to annotate which form he or she preferred for each anaphor position on the sheets.</Paragraph>
      <Paragraph position="1"> We selected five texts generated by our system for the test. The numbers of clauses in the texts are 5, 12, 12, 21 and 34; the numbers of anaphors in the texts are 4, 11, 11, 20 and 34.</Paragraph>
      <Paragraph position="2"> See the Appendix for the first three test texts.</Paragraph>
      <Paragraph position="3"> For convenience, we summarise the occurrence of anaphors in the test texts in a graphical form in Fig. 4. In the figure, each box represents a clause and at the right end is the accompanying punctuation mark. Each box is divided into three parts which represent the topic, the sub-ject and the direct object positions of the clause. The numbers in a box, except for the first occurrences in the text, are the indices of anaphors in the corresponding clauses. Initial references are indicated by bold italics. For example, in Text 2, the numbers 1, 2, 3 and 4 occurring in the first, 5th, 8th and lOth clauses, respectively, are initial references; others are anaphors.</Paragraph>
      <Paragraph position="4"> After the annotations were collected, we carried out comparisons between the speakers' results and the generated texts to investigate the performance of the test rules. In each comparison, we noted down the number of matches between the computer generated text and the human result. In the following, we use Cij to denote the text indexed j generated by the system equipped with Rule TRi, where i is 1 to 3 and j is 1 to 5; and Hkt to denote the resulting text indexed l of speaker k, where k is 1 to 12 and l is 1 to 5. The comparison work is summarised procedurally as below.</Paragraph>
      <Paragraph position="5"> for each rule TRi for each speaker j for each text k compare Cik with Hjk and note down the number of matches of anaphors between them</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>