<?xml version="1.0" standalone="yes"?>
<Paper uid="C00-1082">
  <Title>Bunsetsu Identification Using Category-Exclusive Rules</Title>
  <Section position="4" start_page="568" end_page="570" type="evalu">
    <SectionTitle>
4 Experiments and discussion
</SectionTitle>
    <Paragraph position="0"> In our experiments we used a Kyoto University text eorlms (Knrohashi and Nagao, 1997), which is a tagged corpus made Ul) of articles fi'om the Mainichi newspaper. All exl)eriments reported in this paper we.re performed using art, ielcs dated fi'om ,\]mmary \] to 5, 1995. We obtained the correct infi)rnmtion ()n morphoh)gy and }mnse.t;su identiticathm from the tagged corpus.</Paragraph>
    <Paragraph position="1"> The following experiments were conducted to determine which supervised \]earnillg~ lnethod achieves the high&lt;'.st a(:Cllra(:y l~tl;e.</Paragraph>
    <Paragraph position="2">  ltecm&gt;e we used F, xlmriment \] in maki,lg Method I and Method 2, \]i;Xl)erinieut 71 is a ch)sc'd data..~et for Mel:l~od \] and Method 2. So, we l)crformed Exlmriment 2.</Paragraph>
    <Paragraph position="3"> The ,'(;suits arc. listed in '12fl)lt;,q I to d. \Ve used KNP2.0b4 (Kurohashi, 11997) mM KNP2.0t/6 (Kurohashi: 1998), which are bmlsetsu identitication and syntael;i(&amp;quot; analysis systems using tmmy hand-made rules in addition 1;o the six methods des(:ribed in Section 3. Be('mtse KNP is not based on a machine learning inethod but :many hand-made rules, in the KNP results &amp;quot;Learning selY and '~'.Test set&amp;quot; in the tallies have nt) meanings. In the eXll(wiment of KNP, we also uses morphological information in a corpus.</Paragraph>
    <Paragraph position="4"> ~\].~hc ';F': ill l;\]le tables indicates the F-measure~ which is the. harmonic mean of a recall and a precision. A recall is l;he fl'action of correctly identilied partitions out of all the partitions. A t)reeision is the ffaeth)n of correctly identitied partitions out of all the SlmCeS which were judged to have a partition mark inserted.</Paragraph>
    <Paragraph position="5"> Tables I to -/I show the. following results: (r) In the test set I;he dc.cision-tree method was a little better thmt the maximmn-entropy  The nmnber of&amp;quot; spaces between two morphemes is 32,3o4. The number of partitions is 11,756.</Paragraph>
    <Paragraph position="6"> method. Although the maximuln-entropy method has a weak point in that it, does not learn the combinations of features, we could overcome this weakness by malting almost all of the combilmtions of features to produce a higher accuracy rate.</Paragraph>
    <Paragraph position="7"> * Tile decision-list nlethod was better titan the maximum-entropy method in this experinmnt.</Paragraph>
    <Paragraph position="8"> * Tile example-based nlethod obtained the highest accuracy rate among the four existing methods. null * Alttlough Method 1, which uses tim category-exclusive rule, was worse than the exmnple-based method, it was better than tile decision-list method. One reason for this was that tile decision-list metllod chooses rules rmldomly when multiple rules have identical probabilities mid fl'equeneies.</Paragraph>
    <Paragraph position="9"> * Method 2, which uses the category-exchlsive rule with the highest similarity, achieved the highest accuracy rate among tile supervised learning methods.</Paragraph>
    <Paragraph position="10"> * Tim example-based method, tim decision-list inethod, Method 1 and Method 2 obtained accuracy rates of about 100% for the leanfing set. This indicates that these methods m:e especially strong for learning sets.</Paragraph>
    <Paragraph position="11"> * Tile two methods using similarity example-based method mid Method 2) were always better than the other methods, indicating that the use of similarity is eflective if we can define it approl)riately.</Paragraph>
    <Paragraph position="12"> * We carried out experinmnts by using KNP, a system that uses ninny ha.nd-made rules. The F-measure of KNP was highest in the test set.</Paragraph>
    <Paragraph position="13"> * We used two versions of KNP, KNP 2.0b4 and KNP 2.0b6. The latter was mudl better tlmn tlm former, iudicating tha.t the improvements made by hand are effective. But, the maintenance of rules by hand has a limit, so the improvements made by hand are not always effective. null Tlle above experiments indicate that Method 2 is best among the machine learning methods '5.</Paragraph>
    <Paragraph position="14"> In Table 5 we show some cases which were partitioned incorrectly with KNP but correctly with 51n these experiments, the. differences were very small. But, we think that the differences are significant to some extent because we performed Experiment 1 and Experiment 2, the data we used are a large corplls containing about a few ten thousand morphemes and tagged objectively in advance, and the difference of about 0.1% is large in the precisions of 99%.</Paragraph>
    <Paragraph position="15">  L_{&amp;quot;&amp;quot; 1)e patient with ... steadily) lyoyuu wo \] motte \]~xrT;~l~ shirizoke i(enough Stl'engi;h) obj (have) (1)eat off) (... beat off ... having enough sl;rength) ~aisha wo I gurupu-wake \[~ ,.o.,v,,,,y obj (~r,,,,pi,,~) (do) (... 11o grouping companies) Method 2. A partition with &amp;quot;NEED&amp;quot; indicates that  KNP missed inserting the i)artition mark, and a partition with &amp;quot;WRONG&amp;quot; indicates that KNP inserted the partitiol~ mark incorrectly. In the test set of Experiment 1, the F-measure of KNP2.0b6 was 99.66%. The F-measur(. ~ increases to 99.83%, ml(ler the assumption that when KNP2.0t)6 or Method 2 in correct, the answer is correct. Although the accuracy rate for KNP2.0b6 was high, there were some cases in which KNP t)artitioned incorrectly and Method 2 partitioned correctly, A combination of Method 2 with KNP2.0b6 may be able to iml)rove the Flile~lsllrO. null The only 1)revious research resolving Imnnetsu identification by machine learning methods, in the work by Zhang (Zhang and Ozeki, 1998). The decision-tree, ine, thod was used in this work. But this work used only a small mmther of intorlll;ttion for t)llllsetsll identification&amp;quot; and (lid not achieve lligh accuracy rat;es. (The recall rate was 97.6%(=2502/(2502+62)), the 1)recision rate was 92.4%(=2502/(2502+205)), and F-measure was 94.2%.)</Paragraph>
  </Section>
class="xml-element"></Paper>