<?xml version="1.0" standalone="yes"?>
<Paper uid="W06-3324">
  <Title>A Pragmatic Approach to Summary Extraction in Clinical Trials</Title>
  <Section position="2" start_page="124" end_page="124" type="abstr">
    <SectionTitle>
2 Evaluation
</SectionTitle>
    <Paragraph position="0"> Manual PEA validation was done on a random sample of 300 trials. For a stricter test, the 13,110 studies with Purpose sections short enough to include in full without any type of processing or decision were not part of the random sample.</Paragraph>
    <Paragraph position="1"> Judgments were provided by the authors, one of whom was not involved in the development of PEA code. The 300 English extracts (before translation) were compared against the full-text Purpose sections in the clinical trials, with compression rate averaging 30%. Evaluation was done on a 3-point scale: perfect extraction, appropriate, wrong text.</Paragraph>
    <Paragraph position="2"> Inter-annotator agreement using Cohen's kappa was considered to be good (Kappa = 0.756987).</Paragraph>
    <Paragraph position="3"> Table 2 shows evaluation results after inter-rater differences were reconciled:</Paragraph>
  </Section>
class="xml-element"></Paper>