<?xml version="1.0" standalone="yes"?>
<Paper uid="P05-1035">
  <Title>QARLA:A Framework for the Evaluation of Text Summarization Systems</Title>
  <Section position="1" start_page="0" end_page="0" type="abstr">
    <SectionTitle>
Abstract
</SectionTitle>
    <Paragraph position="0"> This paper presents a probabilistic framework, QARLA, for the evaluation of text summarisation systems. The input of the framework is a set of manual (reference) summaries, a set of base-line (automatic) summaries and a set of similarity metrics between summaries.</Paragraph>
    <Paragraph position="1"> It provides i) a measure to evaluate the quality of any set of similarity metrics, ii) a measure to evaluate the quality of a summary using an optimal set of similarity metrics, and iii) a measure to evaluate whether the set of baseline summaries is reliable or may produce biased results.</Paragraph>
    <Paragraph position="2"> Compared to previous approaches, our framework is able to combine different metrics and evaluate the quality of a set of metrics without any a-priori weighting of their relative importance. We provide quantitative evidence about the effectiveness of the approach to improve the automatic evaluation of text summarisation systems by combining several similarity metrics.</Paragraph>
  </Section>
class="xml-element"></Paper>