<?xml version="1.0" standalone="yes"?>
<Paper uid="W06-0702">
  <Title>Challenges in Evaluating Sumaries of Short Stories</Title>
  <Section position="7" start_page="14" end_page="14" type="concl">
    <SectionTitle>
4 Conclusions and Future Work
</SectionTitle>
    <Paragraph position="0"> This paper presented an experimental way of evaluating automatically produced summaries of literary short stories.</Paragraph>
    <Paragraph position="1"> In the course of our experiments we have remarked a few isues pertinent to evaluating summaries of short fiction.</Paragraph>
    <Paragraph position="2"> Firstly, higher degree of redundancy of sentences in texts makes measures based on sentence overlap not very enlightening when evaluating extracted summaries. Secondly, at least in our corpus, the sentence overlap-based measures do not correlate well with those measuring opinions of humans about summaries.</Paragraph>
    <Paragraph position="3"> This work is exploratory, and as such raises more questions than it answers. In order to evaluate summaries of literary works in a meaningful and reliable way one needs to define criteria which make such summaries suitable or not suitable for a particular purpose. We wil explore this isue in our future work. We also intend to apply the pyramid method of evaluating summaries to extracted summaries produced by the human annotators.</Paragraph>
  </Section>
class="xml-element"></Paper>