<?xml version="1.0" standalone="yes"?>
<Paper uid="P06-1108">
  <Title>Event Extraction in a Plot Advice Agent</Title>
  <Section position="4" start_page="857" end_page="857" type="intro">
    <SectionTitle>
2 Corpus
</SectionTitle>
    <Paragraph position="0"> In order to train our agent, we collected a corpus of 290 stories from primary schools based on two different exemplar stories. The first is an episode of &amp;quot;The Wonderful Adventures of Nils&amp;quot; by Selma Lagerloff (160 stories) and the second a re-telling of &amp;quot;The Treasure Thief&amp;quot; by Herodotus (130 stories). These will be referred to as the &amp;quot;Adventure&amp;quot; and &amp;quot;Thief&amp;quot; corpora.</Paragraph>
    <Section position="1" start_page="857" end_page="857" type="sub_section">
      <SectionTitle>
2.1 Rating
</SectionTitle>
      <Paragraph position="0"> An experienced teacher, Rater A, designed a rating scheme equivalent to those used in schools. The scheme rates the stories as follows:  1. Excellent: An excellent story shows that  the student has &amp;quot;read beyond the lines&amp;quot; and demonstrates a deep understanding of the story, using inference to grasp points that may not have been explicit in the story. The student should be able to retrieve all the important links, and not all the details, but the right details.</Paragraph>
      <Paragraph position="1">  2. Good: A good story shows that the student understood the story and has &amp;quot;read between the lines.&amp;quot; The student recalls the main events  and links in the plot. However, the student shows no deep understanding of the plot and does not make use of inference. This can often be detected by the student leaving out an important link or emphasizing the wrong details. null  3. Fair: A fair story shows that student has  listened to the story but not understood the story, and so is only trying to repeat what they have heard. This is shown by the fact that the fair story is missing multiple important links in the story, including a possibly vital part of the story.</Paragraph>
      <Paragraph position="2">  4. Poor: A poor story shows the student has had  trouble listening to the story. The poor story is missing a substantial amount of the plot, with characters left out and events confused. The student has trouble connecting the parts of the story.</Paragraph>
      <Paragraph position="3"> To check the reliability of the rating scheme, two other teachers (Rater B and Rater C) rated subsets (82 and 68 respectively) of each of the corpora. While their absolute agreement with Rater A  makes the task appear subjective (58% for B and 53% for C), their relative agreement was high, as almost all disagreements were by one level in the rating scheme. Therefore we use Cronbach's a and tb instead of Cohen's or Fleiss' k to take into account the fact that our scale is ordinal. Between Rater A and B there was a Cronbach's a statistic of .90 and a Kendall's tb statistic of .74. Between Rater B and C there was a Cronbach's a statistic of .87 and Kendall's tb statistic of .67. These statistics show the rating scheme to be reliable and the distribution of plot ratings are given in Table 1.</Paragraph>
    </Section>
    <Section position="2" start_page="857" end_page="857" type="sub_section">
      <SectionTitle>
2.2 Linguistic Issues
</SectionTitle>
      <Paragraph position="0"> One challenge facing this task is the ungrammatical and highly irregular text produced by the students. Many stories consist of one long run-on sentence. This leads a traditional parsing system with a direct mapping from the parse tree to a semantic representation to fail to achieve a parse on 35% percent of the stories, and as such could not be used (Bos et al., 2004). The stories exhibit frequent use of reported speech and the switching from first-person to third-person within a single sentence. Lastly, the use of incorrect spelling e.g., &amp;quot;stalk&amp;quot; for &amp;quot;stork&amp;quot; appearing in multiple stories in the corpus, the consistent usage of homonyms such as &amp;quot;there&amp;quot; for &amp;quot;their,&amp;quot; and the invention of words (&amp;quot;torlix&amp;quot;), all prove to be frequent.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>