<?xml version="1.0" standalone="yes"?>
<Paper uid="P03-2037">
  <Title>Automatic Detection of Grammar Elements that Decrease Readability</Title>
  <Section position="5" start_page="0" end_page="0" type="concl">
    <SectionTitle>
4 Experiment
</SectionTitle>
    <Paragraph position="0"> We conducted two experiments, in order to check the performance of our detector.</Paragraph>
    <Paragraph position="1">  The first test is a closed test, where we examine whether grammar elements in example sentences of TCS are detected correctly. TCS gives 840 example sentences, and there are 802 sentences from which their grammar elements are detected correctly. From the rest 38 sentences, our detector failed to detect the right grammar element. This result shows that our program achieves the sufficient recall 95% in the closed test. Almost of these errors are caused failure of morphological analysis.</Paragraph>
    <Paragraph position="2"> The second test is an open test, where we examine whether grammar elements in example sentences of the textbook, which is written for learners preparing for the Japanese Language Proficiency Test (Tomomatsu et al., 1996), are detected correctly. The textbook gives 1110 example sentences, and there are 680 sentences from which their grammar elements are detected correctly. Wrong grammar elements are detected from 71 sentences, and no grammar elements are detected from the rest 359 sentences. So, the recall of automatic detection of grammar elements is 61%, and the precision is 90%. The major reason of these failures is strictness of several rules; several rules that are generated from example pairs automatically are overfitting to example pairs so that they cannot detect variations in the textbook. We think that relaxation of such rules will eliminate these failures.</Paragraph>
  </Section>
class="xml-element"></Paper>