<?xml version="1.0" standalone="yes"?>
<Paper uid="W02-0405">
  <Title>Using Summaries in Document Retrieval</Title>
  <Section position="9" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
6 Results
</SectionTitle>
    <Paragraph position="0"> Thirty queries were applied first to full-text and then limited to LEAD only. The results of each approach were evaluated twice, once from the all reference evaluation scope and once from the highly relevant reference evaluation scope.</Paragraph>
    <Paragraph position="1"> For the customer perspective that Searchable LEAD targets - the highly relevant reference evaluation scope - there was a sizeable improvement in answer set precision, which increased an average of .286, from an average of .230 to an average of .516 when the query was limited to the LEAD. As Table 2 also shows, recall decreased an average of .192 across the thirty queries. The average standard f-measure across the thirty queries increased .150, from  highly relevant reference evaluation scope.</Paragraph>
    <Paragraph position="2"> (NOTE: The f-measure listed under LEAD is lower than both the corresponding recall and precision. Keep in mind that the figures above represent averages for thirty queries. The f-measure .450 thus is not based on a recall rate of .593 and a precision rate of .516 but rather it is the average of thirty individual f-measures. This also explains f-measures provided in Tables 3 and 4.) When evaluated from the perspective of customers who want to retrieve all references to a topic, restricting the query to the LEAD on average resulted in a substantial drop in recall, from an average of .704 to an average of .232, a drop of .472 on average. The small .082 increase in average precision rates barely provides any offsetting benefits, as Table 3 shows. Average f-measures across the thirty queries dropped .324. Customers who want to retrieve all references not surprisingly do not benefit at all from using Searchable LEAD.</Paragraph>
    <Paragraph position="3">  The trends represented by the results in these tables were generally consistent with the results for individual queries. When using the highly relevant reference evaluation scope, precision rates and f-measures increased for 26 of the thirty queries when shifting from full-text to LEAD. When using the all reference evaluation scope, recall rates decreased or stayed steady and f-measures decreased for all thirty queries when shifting from full-text to LEAD.</Paragraph>
    <Paragraph position="4"> For all queries tested, the number of documents retrieved when using Searchable LEAD not surprisingly was lower than when using full-text. Searchable LEAD-based answer sets on average were one fourth the size of fulltext-based answer sets, 50.6 documents vs. 198.7 documents, respectively.</Paragraph>
  </Section>
class="xml-element"></Paper>