<?xml version="1.0" standalone="yes"?>
<Paper uid="C00-1081">
  <Title>A Stochastic Parser Based on a Structural Word Prediction Model Shinsuke MORI, Masafumi NISHIMURA, Nobuyasu ITOH,</Title>
  <Section position="5" start_page="560" end_page="562" type="evalu">
    <SectionTitle>
4 Evaluation
</SectionTitle>
    <Paragraph position="0"> We developed a POS-based model and its lexicalized version explained in section 2 to evaluate their predictive power, and implemented parsers based on them that calculate the most probable dependency tree fi'om a given character sequence, using the solution search algorithm explained in section 3 to observe their accuracy. In this section, we present and discuss the experimental results.</Paragraph>
    <Section position="1" start_page="560" end_page="562" type="sub_section">
      <SectionTitle>
4.1 Conditions on the Experiments
</SectionTitle>
      <Paragraph position="0"> The corpus used in our experiments consists of articles extracted from a financial newspaper (Nihon  Keizai ,%inbun). Each sentence in tile articles is segmented into words and its dependency structure is annotated by linguists using an editor specially designed for this task at our site. The corpus was divided into ten parts; the parameters of the model were estimated fi:om nine of them and the model was tested on the rest (see Table 1). A small part of each leaning corpus is withheld from parameter estimation and used to select the words to be lexicalized. After checking the learning corpus, the maximum number of partial parse trees is set to 10 To evaluate the predictive power of our model, we calculated their cross entropy on the test corpns. In this process, the annotated tree in the test corpus is used as the structure of the sentences. Therefore the probability of each sentence in the test corpus is not the summation over all its possible derivations. To compare the POS-based model and the \]exicalized model, we constructed these models using the same learning corpus and calcnlated their cross entropy on the same test corpus. The POS-based model and the }exicalized model have the same mfl~nown word model, thus its contribution to the cross entropy is constant.</Paragraph>
      <Paragraph position="1"> We implemented a parser based on the dependency models. Since our models, inchsding a character-l)ased unknown word model, can return the best parse tree with its probability for any input, we can build a parser that receives a character sequence as input. It is not easy to evaluate, however, because errors may occur in segmentation of the sequence into words and in estimation of their POSs. For this reason, in the tbllowing description, we assume a word sequence as the input.</Paragraph>
      <Paragraph position="2"> The criterion for a parser is the accuracy of its output dependency relations. This criterion is widely used to evahmte Japanese dependency parsers. The accuracy is the ratio of the nnmber of the words a.nnotated with the same dependency to the numl)er of the words as in the corpus: accuracy =#=words dependiug on tilt correct word ~words Tile last word and the second-to-last word of&amp;quot; a sentence are excluded, because there is no ambiguity. The last word has no word to depend on and the second-todast word depends always on the last word.</Paragraph>
    </Section>
    <Section position="2" start_page="562" end_page="562" type="sub_section">
      <SectionTitle>
4.2 Ewduation
</SectionTitle>
      <Paragraph position="0"> Table 2 shows the cross entropy and parsing accuracy Of the baseline, where all words depend on the next word, the POS-based dependency model and two lexicalized dependency models. In the selectively lexicalized model, words to be lexicalized are selected by the aJgo,:ithm described in section 2. In the completely lexicalized model, all words arc lcxicalized. This result attests experimentally that the pa.rser based on the selectively lexicalized model is the best parser. As for predictive power, however, the completely lexica.lized model has the lowest cross e~/tropy. Thus this model is estimated to be the best language model for speech recognition. Although there is no direct relation between cross entropy of l;he language model and error ra.te of a speech recognizer, if we consider a spoken la.nguage parser, it ma.y be better to select the words to be lexicalized using other criterion.</Paragraph>
      <Paragraph position="1"> We calculated the cross entropy and the parsing accuracy (if' the model whose parameters arc estimated fi'om ;I/d, 1/16, and 1/64 of the learning corpus. The relation between the learning corpus size and the cross entrol)y or the l)arsing a.ccm:acy is shown in Figure d. The cross entropy has a stronger tendency to decrease as the corpus size increases.</Paragraph>
      <Paragraph position="2"> As for accuracy, there is aJso a tendency for parsers to become more accurate as the size of the learning increases. The size of the cor/)us we h~we all this stage is not at all large, ltowever, its accuracy is at the top level of Japanese parsers, which nse 50,0001.90,000 sentences. Therefore, we conclude that our approach is quite promising.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>