<?xml version="1.0" standalone="yes"?>
<Paper uid="W97-0113">
  <Title>Data Reliability and Its Effects on Automatic Abstracting</Title>
  <Section position="5" start_page="124" end_page="124" type="concl">
    <SectionTitle>
4. CONCLUSION
</SectionTitle>
    <Paragraph position="0"> We have seen how human reliability can affect the performance of automatic abstracting.</Paragraph>
    <Paragraph position="1"> Reliability refers to reproducibility or inter-coder consistency of data, which is measured by the kappa statistic, a metric standardly used in the behavioral sciences. It was found that reliability enhances the strength of &amp;quot;good&amp;quot; attributes for a sentence, leading to an improved performance of abstracting models. But we did not discuss an important question of whether the kappa statistic serves as a general tool for distinguishing &amp;quot;good&amp;quot; from &amp;quot;bad&amp;quot; data for training a learning algorithm.</Paragraph>
    <Paragraph position="2"> We have also found that a set of attributes vary in effectiveness from one text type to another, though texts under consideration are all ~om the same domain. But at the moment, it is not clear to us what is a good attribute for representing texts like columns, for which the abstracting model was found not effective. It could be the case that no good attribute exists for columns. In fact humans are not doing well on them either.</Paragraph>
  </Section>
class="xml-element"></Paper>