<?xml version="1.0" standalone="yes"?>
<Paper uid="W00-1403">
  <Title>An Empirical Study in Multilingual Natural Language Generation: What Should A Text Planner Do?</Title>
  <Section position="4" start_page="33" end_page="33" type="intro">
    <SectionTitle>
3 Experiment
</SectionTitle>
    <Paragraph position="0"> In order to assess how similar discourse structures are across languages, we built manually a corpus of discourse trees for 40 Japanese texts and their corresponding translations. The texts, selected randomly from the ARPA corpus (White and O'Connell, 1994), contained on average about 460 words. We developed a discourse annotation protocol for ,Japanese and English along the lines followed by Marcu et al. (1999). We used Marcu's discourse annotation tool (1999) in order to manually construct the discourse structure of all Japanese and English texts it, the corpus. 10~. of the Japanese and English texts were rhetorically labeled by two of us. The agreement was statistically significant (Kappa = 0.65.0 &gt; 0.01 for Japanese and Kappa = 0.748,0 &gt; 0.01 for English (Carletta, 1996; Siegel-and Castellan, 1988)). The tool and the annotation protocol are available at. http://www, isi.edt,/~r, zarcu/softwa,-e/. For each pair of Japanese-English discourse, structures, we also built, manually an alignment file, which specified the correspondence between the edus of the Japanese and English texts.</Paragraph>
    <Paragraph position="1"> Using labeled recall and precision figures, we computed the similarity between English and Japanese discourse trees with respect t,o their assignment of edu boundaries, hierarchical spans, nuclearity, and rhetorical relations, Because the trees we comparod differ from one language to the other ill the ntnnber of elernent ary units, the order, of these units, and the way the units are grouped rectirsively into discourse spans, we comptlted two types of recall and precision , :+when :~e.J~laa~tese:~-and.~En~lish::spans -appeared-in the same position linearly. For example, the English tree in Figure 2 is characterized by 10 sub-sentential spans, which span across positions \[1,1\], \[2,2\], \[3,3\], \[4,4\], \[5,5\], \[6,6\], \[1,2\], \[4,5\], \[3,5\], and \[1,5\]. (Span \[1,6\] subsumes 2 sentences, so it is not sub-sentential.) The Japanese discourse tree has only 4 spans that could be matched in the same positions with English spans, namely spans \[1,2\]. \[4,4\], \[5,5\], and \[1,5\]. Hence the similarity between the Japanese tree and the English tree with respect to their discourse structure below the sentence level has a recall of 4/10 and a precision of 4/ll (in Figure 2, there are 11 sub-sentential Japanese spans).</Paragraph>
    <Paragraph position="2"> In computing Position-Independent (P-I) recall and precision figures, even when a Japanese span &amp;quot;floated&amp;quot; during the translation to a position in the English tree that was different from the position in the initial tree, the P-I recall and precision figures are affected less than when computing Position-Dependent figures. The position-independent figures reflect the intuition that if two trees tl and t2 both have a subtree t, tl and 12 are more similar than if they were if they didn't share ally subtree.</Paragraph>
    <Paragraph position="3"> For instance, for the spans at the sub-sentential level in the trees in Figure 2 the position-independent recall is 6/10 and the position-independent precision is 6/11 because in addition to spans \[1,2\], \[4,4\], \[5,5\], and \[1,5\], one can also match Japanese spat, \[1,1\] to English spa,, \[2,2\] and Japanese spa,, \[2,2\] to Japanese span \[1,1\]. The Position-Independent figures offer a more optimistic metric for comparing discourse trees. They span a wider range of values than the Position-Dependent figures, which enables a finer grained comparison, which in turn enables a better characterization of the differ.ences between Japanese and English discourse structures.</Paragraph>
    <Paragraph position="4"> In order to provide a better estimate of how close two discourse trees were, we computed Position-Dependent and -Independent recall and precision figures for the sentential level (where units are given by edus and spans are given by sets of edus or single sentences); paragraph level (where units are given by sentences and spans are given by sets of sentences or single paragraphs): and text level (where units are given by paragraphs and spans are given by sets of paragraphs). These figures offer a detailed picture of how discourse structures and relations are mapped -from one languageto the other. Some of the differences at the sentence level can be explained by differences between the syntactic structures of Japanese  and English. The differences at the paragraph and text levels have a purely rhetorical explanation.</Paragraph>
    <Paragraph position="5"> As expected, when one computes the recall and precision figures with respect to the nuclearity and relation assignments, one also factors in the nuclearity status and the rhetorical relation that is associated with each span.</Paragraph>
    <Paragraph position="6"> Table 1 summarizes the results (P-D and P-I (R)ecall and (P)recision figures) for each level (Sentence, Paragraph, and Text). It presents Recall and Precision figures with respect to span assignment, nuclearity status, and rhetorical relation labeling of discourse spans. The numbers in the &amp;quot;Weighted Average&amp;quot; line report averages of the Sentence-, Paragraph-, and Text-specific figures, weighted according to the number of units at each level. The numbers in the &amp;quot;All&amp;quot; line reflect recall and precision figures computed across the entire trees, with no attention paid t.o sentence and paragraph boundaries.</Paragraph>
    <Paragraph position="7"> Given the significantly different syntactic structures of Japanese and English. we were not surprised by tile low recall and precision results that reflect the similarity between discourse trees built below the sentence level. However, as Table 1 shows, there are astonishing differences between discourse trees at the paragraph and text. levels as well. For exampie, the Position-Independent figures show that only about 62% of the sentences: and only :about 53% of the hierarchical spans built across sentences could be matched between the two corpora. When one looks at the nuclearity status and rhetorical relations associated with the spans built across sentences, the P-I recall and precision figures drop to about 43c2~ and :/5~ respectively.</Paragraph>
    <Paragraph position="8"> The differences in recall and precision are exl)lained both by differen,-es in the way information is packaged rote paragraphs in the-two languages arid the way it is structured rhetorically both within and above the paragraph level.</Paragraph>
    <Paragraph position="9"> 4 How should a multilingual text planner work? The results in Section 3 strongly suggest that if one is to build text plans in the context of a Japanese-English multilingual generation system, a language-independent text planning module whose output is mapped straightforwardly into sentence plans (Iordanskaja et al., 1992; Goldberg et al., 1994) will not do. The differences between the rhetorical structures of Japanese and English texts are simply too big to support the derivation of a unique text plan, which would subsume both the Japanese- and English-specific realizations. If we are to build MGEN systems capable of generating rich texts in languages as distant as English and Japanese, we would need to use more sophisticated techniques. In the rest of this section, we discuss a set of possible approaches, which are consistent with work that has been carried out to date in the NLG field.</Paragraph>
    <Section position="1" start_page="33" end_page="33" type="sub_section">
      <SectionTitle>
4.1 Use text plan representations that are
</SectionTitle>
      <Paragraph position="0"> more abstract than discourse trees Delin et al. (1994) have shown that although tile rhetorical renderings in Figure 1 are non-isomorphic. dmy are alt subsumed by one .commol~, more.abstract t.ext.-plan representation language that forrealizes the procedural relations of Generation and Enablement (Goldman, 1970). One caa~ conceive of.</Paragraph>
      <Paragraph position="1"> text plans being represented as sequences of actions or hierarchies of actions and goals over which one can identify Generation and Enablement relations that hold between them. In such a framework, text planning is carried out ill a language-independent manner. which is then followed by a rhetorical &amp;quot;'fleshing out&amp;quot;. (Delin et al. (1994) have shown how Generation and Enablenlent relations are realized rhetorically in various languages using relations such as PURPOSE, 'SEQUENCE, CONDITION, and MEANS.) Bateman and Rondhuis (1997) suggest that the variability present in Delin et al.'s Rhetorical Struc- null ture analyses in Figure 1 can be explained by the inadequate mixture of intentional and semantic relations, at different levels of granularity. They propose that discourse phenomena should be accounted for at a more abstract level than RST relations and they present a classification system in terms discourse-tree rewriting module capable of rewriting P-specific discourse structures into O-specific discourse structures. When generating texts in language P, the MGEN system works as a monolingum generator. When generating texts in language O, the MGEN system generates a text plan in lanof &amp;quot;stratification&amp;quot;, ..'!mePSafunction?., ,,and .::p~radig,: ........ guage.-~, xnapsitdr~to.=taaag,uageO.,~ anti,then ~proceeds - .. matic/syntagmatic axiality&amp;quot; that enables one to represent discourse structures at multiple levels of abstraction. null Adopting such an approach could be an extremely rewarding enterprise. Unfortunately, the research of Delin et al. (1994) and Bateman and Rondhuis (1997) cannot be applied yet to unrestricted domains. Generation and Enablement are only two of the abstract relations that can hold between actions and goals. And some texts, such as descriptions, are difficult to characterize only in terms of actions and goals. Building a &amp;quot;complete&amp;quot; taxonomy of such abstract relations and identifying adequate mappings between there relations and rhetorical relations are still open problems.</Paragraph>
    </Section>
    <Section position="2" start_page="33" end_page="33" type="sub_section">
      <SectionTitle>
4.2 Derive a language-independent
</SectionTitle>
      <Paragraph position="0"> discourse structure, and then linearize it RSsner and Stede (1992) and Stede (1999) assume that a discourse representation g la Mann and Thompson imposes no contraints on the linear order of the leaves. For tile purpose of multilingual text planning, one can, hence, assume that a language-independent text planner derives first a language-independent rhetorical structure and then linearizes it, i.e., transforms it to make it language specific. The transformations that RSsner and Stede have applied concern primarily re-orderings of the children of some nodes and re-assignment of rhetorical relation labels. But given, for example, tile significant differences between the discourse structures in Figure 2, it is difficult to envision what the language-independent text plan might look like. It is deftnitely possible to conceive of such a text plan representation. However, the linearization module will need then to be much more sophisticated: it will need to be able to rewrite full structures, re-order constituents, aggregate_across possibly non-adjacent units, etc.</Paragraph>
    </Section>
    <Section position="3" start_page="33" end_page="33" type="sub_section">
      <SectionTitle>
4.3 hnplement a text planning algorithm
</SectionTitle>
      <Paragraph position="0"> for one language only. For all other languages, devise discourse-tree rewriting modules In this approach, the system developer assigns a preferrential status to one of the languages that are to be handled I) 3 ' the MGEN system. Lot's call this language P. The system developer implenlents text planning algorithms only for this language. For any other language O, the developer itnplements a  further with the sentence planning and realization stages. Marcu et al. (2000) present and evaluate a discourse-tree rewriting algorithm that exploits machine learning methods in order to map Japanese discourse trees into discourse trees that resemble English-specific renderings.</Paragraph>
      <Paragraph position="1"> The advantage of such an approach is that the tree-rewriting modules can be also used in the context of machine translation systems in order to repackage and re-organize the input text rhetorically, to reflect constraints specific to the target language. The disadvantage is that, from an NLG perspective, there is no guarantee that such a system could produce better results than a system that implements language-dependent text planning modules.</Paragraph>
    </Section>
    <Section position="4" start_page="33" end_page="33" type="sub_section">
      <SectionTitle>
4.4 Derive language-dependent text plans
</SectionTitle>
      <Paragraph position="0"> Another viable approach is to acknowledge that text plans vary significantly across languages and, therefore, should be derived by language-dependent planners. To this end, one could use both top-down (How, 1993; Moore and Paris, 1993) and bottom-up (Marcu, t997; Mellish et al., 1998) text planning algorithms. The advantage of this approach is that it has the potential of producing trees that reflect tile peculiarities specific to any language. The disadvantage is that only the text planning algorithms are general: the plan operators and the rhetorical relations they operate with are languagedependent, and hence, more expensive to develop and maintain.</Paragraph>
    </Section>
    <Section position="5" start_page="33" end_page="33" type="sub_section">
      <SectionTitle>
4.5 Discussion
</SectionTitle>
      <Paragraph position="0"> Depending oil tile languages and text genres it operates with, all MGEN system may get away with a language-independent text planner. However, for sophisticated genres and distant languages, implementing a language-independent planner that is straightforwardly'mapped i:nto sentence, plans does not appear to be a felicitous solution. We enumerated four possible alternatives for addressing the text planning problem in an MGEN system. Each of tile approaches has its own pluses and minuses.</Paragraph>
      <Paragraph position="1"> Which will eventually win in large-scale deployable MGEN systems remains an open question.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>