<?xml version="1.0" standalone="yes"?>
<Paper uid="W03-1118">
  <Title>Text Categorization Using Automatically Acquired Domain Ontology</Title>
  <Section position="6" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
6. Experiments
</SectionTitle>
    <Paragraph position="0"> To assess the power of domain identification of ontology, we test the text categorization ability on two different corpora. The ontology of the first experiment is edited manually; the ontology of the second experiment is automatically acquired. And we also conduct an experiment on the effect of human editing of the automatically acquired ontology.</Paragraph>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
6.1 Single Sentence Test
</SectionTitle>
      <Paragraph position="0"> We test 9,143 sentences, edited manually for a QA system. The accuracy is 94%. These sentences are questions in the financial domain. Because the sentence topics are quite focused, the accuracy is very high. See Table 1.</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
6.2 News Clippings Collection
</SectionTitle>
      <Paragraph position="0"> The second experiment that we conduct is news categorization. We collect daily news from China News Agency (CNA) ranging from 1991 to 1999.</Paragraph>
      <Paragraph position="1"> Each news clipping is short with 352 Chinese characters (about 150 words) on the average.</Paragraph>
      <Paragraph position="2"> There are more than thirty domains and we choose 10 major categories for the experiment.</Paragraph>
    </Section>
    <Section position="3" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
6.3 10 Categories News Categorization
</SectionTitle>
      <Paragraph position="0"> Our ten categories are: domestic arts and education (DD), foreign affairs (FA), finance report (FX), domestic health (HD), Taiwan local news (LD), Taiwan sports (LD), domestic military (MD), domestic politics (PD), Taiwan stock markets (SD), and weather report (WE). From each category, we choose the first 100 news clippings as the training set and the following 100 news clippings as the testing set. After data cleansing, the total training set has 979 news clippings, with 27,951 nodes and less than 10,000 distinct words. The training set for which domain ontologies are automatically acquired is shown in Table 2. A partial view of this ontology is in Figure 1.</Paragraph>
      <Paragraph position="1"> The result of text categorization based on this automatically acquired domain ontology is shown in Table 5, which contains the recall and precision for each domain. Note that, without the help of the event structure, the macro average f-score is 85.16%. Even the total number of domain key concepts is less than 10,000 words (instead of 100,000 words in standard dictionary), we can still obtain a good categorization result. With the help of event structure, the macro average f-score is 85.55%.</Paragraph>
    </Section>
    <Section position="4" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
6.4 Human Editing
</SectionTitle>
      <Paragraph position="0"> To verify the refinement method, we conduct an experiment to compare the result of using automatically acquired domain ontology and that of limited human editing (on only one domain ontology). After the training process, we use domain ontologies to classify the training data, and to calculate the global categorization ambiguity factor formula in order to obtain ambiguous event structure pairs as candidates for human editing. For simplicity, we restrict the action of refinement to deletion. It takes a human editor one half day to finish the task and delete 0.62% nodes (172 out of 27,951 nodes). In the testing phase, we select 928 new news clippings as the testing set. Table 3 shows the results from before and after human editing. Due to time constraints, we only edit the part of the ontology that might affect domain DD. The recall and precision of domain DD increase as well as both the average recall and average precision. In addition, the recall of domains having higher correlation with DD, such as PD and FA, decreases. Apparently, the event structures that mislead the categorization system to theses domain have mostly been deleted. The experiment result is very consistent with our intuition.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>