<?xml version="1.0" standalone="yes"?>
<Paper uid="C94-1093">
  <Title>Restructuring Tagged Corpora with Morpheme Adjustment Rules</Title>
  <Section position="2" start_page="0" end_page="569" type="metho">
    <SectionTitle>
1 Introduction
</SectionTitle>
    <Paragraph position="0"> Recently, mm,y natural language processing researchers have concentrated on corpus-based approaches. Linguistic corpora can be classified as word-segmented corpora, part-of-speech tagged corpora, and parsed corpora, tlecause a part-of-speech tagged corpus is the most important corpus, much corpus-based natural language processing research has been performed using part-of-speech tagged corpora. null Ilowever, building a large part-of-speech tagged corpus is very dillicult. It is even more difficult to build a corpus for languages without explicit word boundary characters, such as Japanese. q'herefore, researchers always complain of the scarcity of data in the corpus.</Paragraph>
    <Paragraph position="1"> To solve this data scarcity problem, previous works proposed methods of increasing the productivity of the labor required for building a part-of-speech tagged corpus. \[1\].</Paragraph>
    <Paragraph position="2"> '.l'his paper proposes another method of acquiring large part-of-speech tagged corpora: restructuring tagged corpora by using morpheme adjustment rules. This method assures good use of the sharable part-of-speech tagged corpora that are already awdlable such as the ATR Dialog Database (ADD) \[2; 3\].</Paragraph>
    <Paragraph position="3"> Ideally, these corpora could be used by all researchers and research groups without any modifications, llowever, actual part-of-speech tagged corpora have the following problems: * Diversity of orthography: A word can be spelled in various ways. In Japanese, there are three types of character sets: kanji (i~.~&amp;quot;-), hiragana (~ to ~ ~), and katakana (:~J # :~J C/). Also, people can use these character sets at their discretion.</Paragraph>
    <Paragraph position="4"> * Diversity of word segmentation: Because the Japanese language has no word boundary characters(i.e, blank spaces), there are no standards of word segmentation. A single word in a certain corpus may be considered nmltiple words in other corpora, and vice versa.</Paragraph>
    <Paragraph position="5"> * Diw;rsity of part-of speech systems: There are no standards for part-of-speech systems. It is true that a detailed part-of-speech system can help the application of part-of-speech information, lint the labor required \['or lmilding corpora will continue to increase. This problem is language-independent.</Paragraph>
    <Paragraph position="6"> Diversities of word-segmentation and part-of-speech systems are fatal problems. The simplest way to solve these problems is to perform a morphological analysis on the raw text in the corpus, with no regard to the word-segmentation and part-of-speech infer mation. Uowever, making a high-quality morphological analyzer demands much time and care. Additionally, it is wasteful to ignore the word-segmentation and part-of-speech i,fformation that h~s been acquired with much effort.</Paragraph>
    <Paragraph position="7"> In restructuring tagged corpora with nmrpheme adjustmel,t rules, tim word-segrnentation and part-of-speech inlbrmation of the original corpus is rewrittm,, making good use of the original corpus information. &amp;quot;riffs method is characterized by reduced manual effort. null In the next section, the method of rest, rueturing tagged corpora is described in detail. Section 3 reports the result of an experiment in rewriting the</Paragraph>
    <Section position="1" start_page="569" end_page="569" type="sub_section">
      <SectionTitle>
2.1 Preparation of Training Set
</SectionTitle>
      <Paragraph position="0"> First, sentences for the training set are chosen from the corpus to be rewritten. New word-segmentation and part-of-speech information (morphological information) is given to the sentences by a morphological analyzer or by hand. Consequently, the training set has two sets of morphological information for the same raw text. Figure I shows an example of the training set.</Paragraph>
      <Paragraph position="1"> A large number of training sentences is desirable, but preparing many sentences requires much time and effort. A vmst nmnber of sentences would be required to extend coverage to content words (such as nouns, w'A)s, ete), but flmctionaI words (such as particles, auxiliary verbs, ere) can be covered with a smaller number of sentences.</Paragraph>
      <Paragraph position="2"> ntw text '~!,'q\]O IfI~&amp; .5/~'(&amp;quot; L ~ 5 Z}~ morphological ('/~qJg \[N~- n-com)(tJ: postp-topic)</Paragraph>
      <Paragraph position="4"/>
    </Section>
    <Section position="2" start_page="569" end_page="569" type="sub_section">
      <SectionTitle>
2.2 Extraction of Morpheme Adjust-
ment Rules
</SectionTitle>
      <Paragraph position="0"> The method of extracting morpheme ad.iustment rules from the training set involves finding correspondence between rewriting units and extracting rules for unknown words: 2.2.1. CorrespoiMenees of Rewriting Units In languages without explicit word boundary characters, such as Japanese, a single word in a certain morphological information system may be divided into multiple words (one-to-many correspondence) in other morphological information systems, multipie words may be unified (many-to-one correspondence), or the segmentation of multiple words may be changed (many-to-many correspondence). Figure 2 shows these correspondences.</Paragraph>
      <Paragraph position="1"> We developed an algorithm to find these correspondences (Appendix A). By using this algorithm, morpheme rewriting rules (Figure 3) can be extracted.</Paragraph>
      <Paragraph position="2">  Rewriting rules such as those shown in Figure 3 can rewrite only the words that appeared in the training set,. If the training set is small, the coverage of the rules will be limited. Ilowever, because this n~orpheme adjnstment is a method of rewriting part-of-speech tagged corpora, the treatment of unknown</Paragraph>
      <Paragraph position="4"> words is easier than with an ordinary morphological anaIyzer, because that our method can make good use of the part-of-speech information of the original corpus. Rules for unknown words without word-segmentation changes between two morphological information systems can be extracted automatically from one-to-one correspondence rules in the rewriting rules.</Paragraph>
      <Paragraph position="5"> Rules tbr unknown words with word-segmentation changes can also be extracted automatically by using information concerning the length of the word's characters. For examph'., when a single verb with two characters in a certaiu morphological information system corresponds to two words (verb-stem with one character and verb-inflection with one character) in another morphological information system, the following rewriting rule is extracted.</Paragraph>
      <Paragraph position="6"> 2(verb)--~ l(verb-stem) l(verb-inflection) Figure 4 shows sample rules for unknown words.</Paragraph>
      <Paragraph position="7"> The heuristic knowledge of character sets that an ordinary Japanese morphological analyzer uses (such as &amp;quot;katakana words are usually proper nom,s&amp;quot;, &amp;quot;verb iMleetion words are spelled using hiragana&amp;quot;, etc.) are also available in this nlorpheme adjustment ted&gt; nique.</Paragraph>
      <Paragraph position="8"> C/ r 2.3 Rewriting of Tagged Corpora</Paragraph>
    </Section>
    <Section position="3" start_page="569" end_page="569" type="sub_section">
      <SectionTitle>
2.3.1 Application of Rewriting Ihnns
</SectionTitle>
      <Paragraph position="0"> By applyi,~g the rewriting rules described in the last subsection to the tagged corpus, a lattice structure</Paragraph>
      <Paragraph position="2"> l{owew;r, this ambiguity is not as great as the aml&gt;iguity that occurs in ordinary morphological analysis because our method makes good use of the inform;vtion of the original corpus. Figure 6 shows the lattice structure formed when using the ordinary morphological analysis on the same raw text. Note that the size of this lattice is greater i.hail the size oF the lattice made by our method.</Paragraph>
      <Paragraph position="3">  The last step in restructuring tagged corpora can be considered a lattice search l)rol)lenl. \[U this step, all of the following knowledge sources for anlhiguil, y resolution used in ordinary morphological analysis is also available in our method: * connection matrix * heuristic preferences (longest word preference, minimum phrase preference, etc.) * stochastic preferences (word n-gram, IIMM, etc.) By using these knowledge sources, the most plausible candidate is chosen. In effect, the original corpus is converted to a new eorptls that uses a different morphological information system.</Paragraph>
    </Section>
  </Section>
  <Section position="3" start_page="569" end_page="571" type="metho">
    <SectionTitle>
3 Experiment
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="569" end_page="571" type="sub_section">
      <SectionTitle>
3.1 Experimental Condition
</SectionTitle>
      <Paragraph position="0"> The targets in our experiment are a morl~hological information system for the A'I?II, l)ialog Database\[2; l'\])hls ambiguity mainly conies from l.he difl;Jrence iii partoLspeech granularity between the two morphological infornla\[,ion systenls, 3\] and a morphological infornlation system lbr the unification-based Japanese grammar used in A'I'R's spoken language parser\[4\]. 'l.'hese two morphological information systenls haw; the following characteristics. null * The ATR Dialog 13at.abase was developed as material for analyzing the characteristics of spoken-style Japanese. '\['herefore, the part-of-speech granularity is coarse. Additionally, because the word-.segmentation is based on a morphological and etymological criterion, compound nouns ~md compound words that fluiction as a single auxiliary w~rb (e.g. &amp;quot;-C ~ zj &amp;quot;) are divided into several shorter word units. Ou the other hand, because this database giw;s little consideration to mechanical processing, stems and inflections oF inflectional words are not segmented.</Paragraph>
      <Paragraph position="1"> * The nnification-based Japanese gramnmr has a medium-grained part-.of-speech (pre-ternlinal) system to make it both c\[tleient and easy to maintain\[5\]. Because the objective of the gram-Ynar is to extract the syntactic structures el Japanese sentences automatically and elIiciently, COml~OUnd words that funel.iml as a single word are usually recognized as a single word. On the other hand, steins amt ilHlectious of inllectional words are segl\[lellted for eollvellience ()l' nlechall.ieal processiug.</Paragraph>
      <Paragraph position="2"> The above descriptions show that these nlC/~rl~hological infl)rulation syst.ems differ. The objectiw~ of this experiment is to examine whether our method can adjust the differences between the two niorphological information systems to ~i considerable extent.</Paragraph>
      <Paragraph position="3"> Firsl;, we chose 1,000 sentences fronl the A;I'li, Dialog Database as the training set and provided the morphological information (word-segmentation and lmrl.-of'-speech) of the unification-based Japanese grallilnar. \%Te prepared 350 seiiteliCes as the Lest selb, separate from the training set. 'Phe t,'st sentences were also giw~n the lnorphological infornlal, ion.</Paragraph>
      <Paragraph position="4"> We extracted 1,5;18 r.orrespondeiiees el' rewrithlg unil.s (i.e. rewi'il.hlg rilies) alid &lt;128 rules For Uliknown words. '\]'\]lc.se rllles can l)e used for the \]Ji direetioila\] rewril.ing experiuienl..</Paragraph>
      <Paragraph position="5"> As the kliowh~dge sOtll'&lt;:e in searching lattices, word bigrauis and part-ol-sl)eech I)igralns were trained wil;h the training set. To perform the hi-directional rewriting experinlent, these bigralns were trained in both inorphologieal infornial;ion systenls.</Paragraph>
      <Paragraph position="6"> '\['O eOl'ilpare o/lr niethod with ordinary niorphological analysis, we dew, loped a sinlple stocha.&lt;~tic iiiorphological analyzer that uses the santo bigrams as the knowledge sourc~,s 2. Ilecause this morphological analyzer has been developed for the comparative experiment, it. catlnot inanage unknown words. 'Fherelbre, the rewriting test was performed by using not only the &amp;quot;2 Of Cotlrse, the ordinary nlorphologlcal analyzer can rewrite the corpus Iilllch nitre accurately by tlshlg richer knowledge gOllr('t~s, llowevei', it onlst he llo{ed tilat onl' nlel, ilod itlvlo c;lll tlSf~ 51lch knowl.dgc S(-ItlI'CI~S,  test sentences, but also the training sentences (close experiment) and the sentences having no imknown words (a subset of the test set).</Paragraph>
      <Paragraph position="7"> Table 1 shows the experinlental conditions ill detail. null</Paragraph>
    </Section>
    <Section position="2" start_page="571" end_page="571" type="sub_section">
      <SectionTitle>
3.2 Rewriting of Morphological Infor-
</SectionTitle>
      <Paragraph position="0"> mation The experiment was performed bi-directionally between the morphological information system of the ATt~ Dialog Database (ADD) and the morphological information system of unification-based Japanese grammar.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>