<?xml version="1.0" standalone="yes"?>
<Paper uid="P02-1064">
  <Title>An Empirical Study of Active Learning with Support Vector Machines for Japanese Word Segmentation</Title>
  <Section position="9" start_page="4" end_page="8" type="evalu">
    <SectionTitle>
5 Experimental Results and Discussion
</SectionTitle>
    <Paragraph position="0"> We used the EDR Japanese Corpus (EDR, 1995) for experiments. The corpus is assembled from various sources such as newspapers, magazines, and textbooks. It contains 208,000 sentences. We selected randomly 20,000 sentences for training and  Hiragana and katakana are phonetic characters which represent Japanese syllables. Katakana is primarily used to write foreign words.</Paragraph>
    <Paragraph position="1"> 10,000 sentences for testing. Then, we created examples using the feature encoding method in Section 4. Through these experiments we used the original SVM tools, the algorithm of which is based on SMO (Sequential Minimal Optimization) by Platt (1999). We used linear SVMs and set a missclassification cost BV to BCBMBE.</Paragraph>
    <Paragraph position="2"> First, we changed the number of labeled examples which were randomly selected. This is an experiment on passive learning. Table 2 shows the accuracy at different sizes of labeled examples.</Paragraph>
    <Paragraph position="3"> Second, we changed the number of examples in a pool and ran the active learning algorithm in Section 3.2. We use the same examples for a pool as those used in the passive learning experiments. We selected 1,000 examples at each iteration of the active learning. Figure 4 shows the learning curve of this experiment and Figure 5 is a close-up of Figure 4. We see from Figure 4 that active learning works quite well and it significantly reduces labeled examples to be required. Let us see how many labeled examples are required to achieve 96.0 % accuracy. In active learning with the pool, the size of which is 2,500 sentences (97,349 examples), only 28,813 labeled examples are needed, whereas in passive learning, about 97,000 examples are required. That means over 70 % reduction is realized by active learning. In the case of 97 % accuracy, approximately the same percentage of reduction is realized when using the pool, the size of which is 20,000 sentences (776,586 examples).</Paragraph>
    <Paragraph position="4"> Now let us see how the accuracy curve varies depending on the size of a pool. Surprisingly, the performance of a larger pool is worse than that of a smaller pool in the early stage of training  . One reason for this could be that support vectors in selected examples at each iteration from a larger pool make larger clusters than those selected from a smaller pool do. In other words, in the case of a larger pool, more examples selected at each iteration would be similar to each other. We computed variances  of each 1,000 selected examples at the learning iteration from 2 to 11 (Table 1). The variances of se- null Tong and Koller (2000) have got the similar results in a text classification task with two small pools: 500 and 1000. However, they have concluded that a larger pool is better than a smaller one because the final accuracy of the former is higher than that of the latter.</Paragraph>
    <Paragraph position="5">  lected examples using the 20,000 sentence size pool is always lower than those using the 1,250 sentence size pool. The result is not inconsistent with our hypothesis. null Before we discuss the results of Two Pool Algorithm, we show in Figure 6 how support vectors of a classifier increase and the accuracy changes when using the 2,500 sentence size pool. It is clear that after the accuracy improvement almost stops, the increment of the number of support vectors is down. We also observed the same phenomenon with different sizes of pools. We utilize this phenomenon in Algorithm A.</Paragraph>
    <Paragraph position="6"> Next, we ran Two Pool Algorithm A  . The result is shown in Figure 7. The accuracy curve of Algorithm A is better than that of the previously proposed method at the number of labeled examples roughly up to 20,000. After that, however, the performance of Algorithm A does not clearly exceed that of the previous method.</Paragraph>
    <Paragraph position="7"> The result of Algorithm B is shown in Figure 8.</Paragraph>
    <Paragraph position="8"> We have tried three values for AE : 5 %, 10 %, and 20 %. The performance with AE of 10 %, which is best, is plotted in Figure 8. As noted above, the improvement by Algorithm A is limited, whereas it is remarkable that the accuracy curve of Algorithm B is always the same or better than those of the previous algorithm with different sizes of pools (the detailed information about the performance is shown in Table 3). To achieve 97.0 % accuracy Algorithm B requires only 59,813 labeled examples, while passive as:  and D2 is the number of selected examples. null  In order to stabilize the algorithm, we use the following strategy at (d) in Figure 3: add new unlabeled examples to the primary pool when the current increment of support vectors is less than half of the average increment.</Paragraph>
    <Paragraph position="9">  labeled examples and the previous method with the 200,000 sentence size pool requires 100,813. That means 82.6 % and 40.7 % reduction compared to passive learning and the previous method with the 200,000 sentence size pool, respectively.</Paragraph>
  </Section>
class="xml-element"></Paper>