<?xml version="1.0" standalone="yes"?>
<Paper uid="W97-0904">
  <Title>ESTIMATING THE TRUE PERFORMANCE OF CLASSIFICATION-BASED NLP TECHNOLOGY</Title>
  <Section position="3" start_page="0" end_page="0" type="intro">
    <SectionTitle>
INTRODUCTION
</SectionTitle>
    <Paragraph position="0"> The objective of learning classifications from sample text is to classify and predict successfidly on new text. For example, in developing a system for grading student writing samples, the objective is to learn how to classify student writing samples into grade categories so that we may use the system to predict successfully the grade categories for new samples of student writing (Nolan, 1997a).</Paragraph>
    <Paragraph position="1"> The most commonly used measure of success or failure is a classifier's error rate (Weiss &amp; Kulikowski, 1991). Each .time the classifier is presented with a case, it makes a decision about the appropriate class for the case. Sometimes it is right; sometimes it is wrong.</Paragraph>
    <Paragraph position="2"> The true error rate is statistically defined as the error rate of the classifier on a large number of new cases that converge in the limit to the actual population distribution.</Paragraph>
    <Paragraph position="3"> If we were given an unlimited number of cases, the true error rate could be readily computed as the number of samples approached infinity. In the real world, the number of samples is always finite, and typically relatively small. The major question is then whether it is possible to extrapolate from empirical error rates calculated from small sample results to the true error rate. It turns out that there are a number of ways of presenting sample cases to a classifier to get better estimates of the true error rate. Some of these techniques are better than others. In statistical terms, some estimators of the true error rate are considered biased. They tend to estimate too low, i.e., on the optimistic side, or too high, i.e., on the pessimistic side.</Paragraph>
    <Paragraph position="4"> In the next section, we will define just what an error is when using classification systems for natural language processing. The apparent error rate will be contrasted with the true error rate. The effect of classifier complexity and feature dimensionality on classification results will be followed by conclusions.</Paragraph>
  </Section>
class="xml-element"></Paper>