<?xml version="1.0" standalone="yes"?>
<Paper uid="C02-1064">
  <Title>Text Generation from Keywords Kiyotaka Uchimoto + Satoshi Sekine ++</Title>
  <Section position="6" start_page="3" end_page="3" type="evalu">
    <SectionTitle>
5 Evaluation
</SectionTitle>
    <Paragraph position="0"> To evaluate our system we made 30 sets of keywords, with three keywords in each set, as shown in Table 1. A human subject selected the sets from headwords that were found ten  times or more in the newspaper articles on January 1st in the Kyoto University text corpus (Version 3.0) without looking at the articles. We evaluated each model by the percentage of outputs that were subjectively judged as appropriate by one of the authors. We used two evaluation standards.</Paragraph>
    <Paragraph position="1">  * Standard 1: If the dependency tree ranked first is semantically and grammatically appropriate, it is judged as appropriate.</Paragraph>
    <Paragraph position="2"> * Standard 2: If there is at least one depen- null dency tree that is ranked within the top ten and is semantically and grammatically appropriate, it is judged as appropriate. We used headwords that were found five times or more in the newspaper articles appearing from January 1st to 16th in the Kyoto University text corpus and also found in those appearing on January 1st as the set of headwords, KS. For headwords that were not in KS, we added their major part-of-speech categories to the set. We trained our keyword-production models by using 1,129 sentences (containing 10,201 headwords) from newspaper articles appearing on January 1st. We used a morpheme model and a dependency model identical to those proposed by Uchimoto et al. (Uchimoto et al., 2001; Uchimoto et al., 1999; Uchimoto et al., 2000b). To train the models, we used 8,835 sentences from newspaper articles appearing from January 1st to 9th in 1995. Generation rules were acquired from newspaper articles appearing from January 1st to 16th. The total number of sentences was 18,435.</Paragraph>
    <Paragraph position="3"> First, we evaluated the outputs generated when the rightmost two keywords, such as &amp;quot; anda R,&amp;quot; on each line of Table 1 were input.</Paragraph>
    <Paragraph position="4"> Table 2 shows the results. KM1 through KM5 stand for the five keyword-production models describedinSection4.1,andMMandDMstand for the morpheme and the dependency models, respectively. The symbol + indicates a combination of models. In the models without MM, DM, or both, P(M|T)andP(D|M,T)wereassumed to be 1. We carried out additional experimentswithmodelsthatconsideredboththe null anterior and posterior words, such as the combination of KM1 and KM2 or KM3 and KM4.</Paragraph>
    <Paragraph position="5"> The results were at most 16/30 by standard 1 and 24/30 by standard 1.</Paragraph>
    <Paragraph position="6">  achieved the best results, as shown in Table 2. For models KM1, KM3, and KM5, the results with MM and DM were significantly better than those without MM and DM in the evaluation by standard 1. We believe this was because cases are more tightly connected with verbs than with nouns, so models KM1, KM3, and KM5, which learn the connection between cases and verbs, can better rank the candidate-text sentences that have a natural connection between cases and verbs than other candidates.</Paragraph>
    <Paragraph position="7"> Next, we conducted experiments using the 30 sets of keywords shown in Table 1 as inputs. We used two keyword-production models: model KM3+MM+DM, which achieved the best results in the first experiment, and model KM5+MM+DM, which considers the richest information. We assumed that the input keyword order was appropriate and did not reorder the keywords. The results for both models were the same: 19/30 in the evaluation by standard 1 and 24/30 in the evaluation by standard 2. The right column of Table 1 shows examples of the system output. For example, for the input &amp;quot;R(syourai, in the future),  Frontier Party will be born in the future.&amp;quot;) was generated. This output was automatically complemented by the appropriate modality &amp;quot;iO&amp;quot;(darou, will), which agrees with the word &amp;quot;R&amp;quot;(syourai, in the future), as well as by post-positional particles such as &amp;quot; x&amp;quot;(wa, case marker) and &amp;quot;U&amp;quot;(ga). For the input &amp;quot; (gaikoku-jin, a foreigner),C (kanyuu, to join), and  C(zouka,toincrease)&amp;quot;, the dependency tree &amp;quot;(( w [gaikokujin no]CU[kanyuu sya ga]) C`oM[zouka shite iru] )&amp;quot; (&amp;quot;Foreigner members are increasing in number.&amp;quot;) was generated. This output was complemented not only by the modality expression &amp;quot;`o M&amp;quot;(shite iru, the progressive form) and post-positional particles such as &amp;quot;w&amp;quot;(no,of) and &amp;quot;U&amp;quot;(ga), but also by the suffix &amp;quot;&amp;quot; (sya, person), and a compound noun &amp;quot;C&amp;quot; (kanyuu sya, member) was generated naturally. In six cases, though, we did not obtain appropriate outputs because the candidate-text sentences were not appropriately ranked. Improving the back-off ability of the model by using classified words or synonyms as features should enable us to rank sentences more appropriately.</Paragraph>
  </Section>
class="xml-element"></Paper>