File Information
File: 05-lr/acl_arc_1_sum/cleansed_text/xml_by_section/metho/99/p99-1049_metho.xml
Size: 29,102 bytes
Last Modified: 2025-10-06 14:15:24
<?xml version="1.0" standalone="yes"?> <Paper uid="P99-1049"> <Title>An Efficient Statistical Speech Act Type Tagging System for Speech Translation Systems</Title> <Section position="4" start_page="0" end_page="381" type="metho"> <SectionTitle> 2 The Problems </SectionTitle> <Paragraph position="0"> In this section, we briefly explain our speech act type tags and the tagged data and then formally define the tagging problem.</Paragraph> <Section position="1" start_page="381" end_page="381" type="sub_section"> <SectionTitle> 2.1 Data and Tags </SectionTitle> <Paragraph position="0"> The data used in this study is a collection of transcribed dialogues on a travel arrangement task between Japanese and English speakers mediated by interpreters (Morimoto et al., 1994). The transcriptions were separated by language, i.e., English and Japanese, and the resultant two corpora share the same content. Both transcriptions went through morphological analysis, which was manually checked. The transcriptions have clear turn boundaries (TB's).</Paragraph> <Paragraph position="1"> Some of the Japanese and English dialogue files were manually segmented into speech act units (SA units) and assigned with speech act type tags (SA tags). The SA tags represent a speaker's intention in an utterance, and is more or less similar to the traditional illocutionary force type (Searle, 1969).</Paragraph> <Paragraph position="2"> The SA tags for the Japanese language were based on the set proposed by Seligman et al. (1994) and had 29 types. The English SA tags were based on the Japanese tags, but we redesigned and reduced the size to 17 types. We believed that an excessively detailed tag classification would decrease the inter-coder reliability and so pruned some detailed tags) The following lines show an example of the English tagged dialogues. Two turns uttered by a hotel clerk and a customer were Segmented into SA units and assigned with SA tags.</Paragraph> <Paragraph position="3"> <clerk's turn> Hello, (expressive) New York City Hotel, (inform) may I help you ? (offer) <customer(interpreter)'s turn> Hello, (expressive) my name is Hiroko Tanaka (inform) and I would like to make a reservation for a room at your hotel. (desire) The tagging work to the dialogue was conducted by experts who studied the tagging manual beforehand. The manual described the tag definitions and turn segmentation strategies and gave examples. The work involved three experts for the Japanese corpus and two experts for the English corpus. 2 The result was checked and corrected by one expert for each language. Therefore, since the work was done by one expert, the inter-coder tagging instability was suppressed to a minimum. As the result of the tagging, we obtained 95 common dialogue files with SA tags for Japanese and English and used them in our experiments.</Paragraph> <Paragraph position="4"> 1Japanese tags, for example, had four tags mainly used for dialogue endings: thank, offer-follow-up, goodwishes, and farewell, most of which were reduced to expressive in English.</Paragraph> </Section> <Section position="2" start_page="381" end_page="381" type="sub_section"> <SectionTitle> 2.2 Problem Formulation </SectionTitle> <Paragraph position="0"> Our tagging system assumes an input of a word sequence for a dialogue produced by a speech recognition system. The word sequence is accompanied with clear turn boundaries. Here, the words do not contain any punctuation marks. The word sequence can be viewed as a sequence of quadruples: &quot;'&quot; (Wi-1, li-1, ai-1, si-1), (wi, li, ai, 8i)... where wi represents a surface wordform, and each vector represents the following additional information for wi.</Paragraph> <Paragraph position="1"> li: canonical form and part of speech of wi (linguistic feature) ai: pause duration measured milliseconds after wi (acoustic feature) si: speaker's identification for wi such as clerk or customer (situational feature) Therefore, an utterance like Hello I am John Phillips and ... uttered by a cuslomer is viewed as a sequence like (Hello, (hello, INTER), 100, customer), (I,(i, PRON),0, customer)), (am, (be, BE), 0, customer) ....</Paragraph> <Paragraph position="2"> From here, we will denote a word sequence as W = wl, w2, .. * wi, .. *, Wn for simplicity. However, note that W is a sequence of quadruples as described above.</Paragraph> <Paragraph position="3"> The task of speech act type tagging in this paper covers two tasks: (1) segmentation of a word sequence into the optimal number of SA units, and (2) assignment of an SA tag to each SA unit. Here, the input is a word sequence with clear TB's, and our tagger takes each turn as a process unit. 3 In this paper, an SA unit is denoted as u and the sequence is denoted as U. An SA tag is denoted as e represents t and the sequence is denoted as T. x s a sequence of x starting from s to e. Therefore, represents a tag sequence from 1 to j.</Paragraph> <Paragraph position="4"> The task is now formally addressed as follows: find the best SA unit sequence U and tag sequence T for each turn when a word sequence W with clear TB's is given. We will treat this problem with the statistical model described in the next section.</Paragraph> </Section> </Section> <Section position="5" start_page="381" end_page="382" type="metho"> <SectionTitle> 3 Statistical Model </SectionTitle> <Paragraph position="0"> The problem addressed in Section 2 can be formalized as a search problem in a word graph that holds all possible combinations of SA units in a turn. We take a probabilistie approach to this problem, which formalizes it as finding a path (U,T) in the word graph that maximizes the probability P(U, T I W).</Paragraph> <Paragraph position="1"> This is formally represented in equation (1). This probability is naturally decomposed into the product of two terms as in equation (3). The first probability in equation (3) represents an arbitrary word sequence constituting one SA unit ui, given hj (the history of SA units and tags from the beginning of a dialogue, hj = uJ-l,t j-l) and input W. The second probability represents the current SA unit u i bearing a particular SA tag tj, given uj, hi, and</Paragraph> <Paragraph position="3"> We call the first term &quot;unit existence probability&quot; Ps and the second term &quot;tagging probability&quot; PT.</Paragraph> <Paragraph position="4"> Figure 1 shows a simplified image of the probability calculation in a word graph, where we have finished processing the word sequence of w~ -1 Now, we estimate the probability for the word sequence w~ +p-1 constituting an SA unit uj and having a particular SA tag tj. Because of the problem of sparse data, these probabilities are hard to directly estimate from the training corpus. We will use the following approximation techniques.</Paragraph> <Section position="1" start_page="382" end_page="382" type="sub_section"> <SectionTitle> 3.1 Unit Existence Probability </SectionTitle> <Paragraph position="0"> The probability of unit existence PE is actually equivalent to the probability that the word sequence w~,..., w,+p-1 exists as one SA unit given h i and W (Fig. 1).</Paragraph> <Paragraph position="1"> We then approximate PE by</Paragraph> <Paragraph position="3"> where the random variable Bw=,,~=+l takes the binary values 1 and 0. A value of 1 corresponds to the existence of an SA unit boundary between wx and w=+l, and a value of 0 to the non-existence of an SA unit boundary. PE is approximated by the product of two types of probabilities: for a word sequence break at both ends of an SA unit and for a nonbreak inside the unit. Notice that the probabilities of the former type adjust an unfairly high probability estimation for an SA unit that is made from a short word sequence.</Paragraph> <Paragraph position="4"> The estimation of PE is now reduced to that of P(Bw=,w~+l I hi, W). This probability is estimated by a probabilistic decision tree and we have P(Bw=,Wx+, I hi, W) ~- P(Bw .... +1 I eE(hj, W)), where riPE is a decision tree that categorizes hj, W into equivalent classes (Jelinek, 1997). We modified C4.5 (Quinlan, 1993) style algorithm to produce probability and used it for this purpose. The decision tree is known to be effective for the data sparseness problem and can take different types of parameters such as discrete and continuous values, which is useful since our word sequence contains both types of features.</Paragraph> <Paragraph position="5"> Through preliminary experiments, we found that hj (the past history of tagging results) was not useful and discarded it. We also found that the probability was well estimated by the information available in a short range of r around w=, which is stored in W.</Paragraph> <Paragraph position="6"> Actually, the attributes used to develop the tree were at~X-\]-7* in W' = ~-r+l&quot; *+r * surface wordforms for ~=-~+1, z+r and the pause duration parts of speech for wx_r+l, between wx and w=+l. The word range r was set from 1 to 3 as we will report in sub-section 5.3.</Paragraph> <Paragraph position="7"> As a result, we obtained the final form of PE as</Paragraph> <Paragraph position="9"/> </Section> <Section position="2" start_page="382" end_page="382" type="sub_section"> <SectionTitle> 3.2 Tagging Probability </SectionTitle> <Paragraph position="0"> The tagging probability PT was estimated by the following formula utilizing a decision tree eT- Two functions named f and g were also utilized to extract information from the word sequence in uj.</Paragraph> <Paragraph position="2"> As this formula indicates, we only used information available with the uj and m histories of SA tags in hi. The function f(uj) outputs the speaker's identification of uj. The function g(uj) extracts cue words for the SA tags from uj using a cue word list. The cue word list was extracted from a training corpus that was manually labeled with the SA tags. For each SA tag, the 10 most dependent words were extracted with a x2-test. After converting these into canonical forms, they were conjoined.</Paragraph> <Paragraph position="3"> To develop a statistical decision tree, we used an input table whose attributes consisted of a cue word list, a speaker's identification, and m previous tags.</Paragraph> <Paragraph position="4"> The value for each cue word was a binary value, where 1 was set when the utterance uj contained the word, or otherwise 0. The effect of f(uj), g(uj), and length m for the tagging performance will be reported in sub-section 5.3.</Paragraph> </Section> </Section> <Section position="6" start_page="382" end_page="383" type="metho"> <SectionTitle> 4 Search Method </SectionTitle> <Paragraph position="0"> A search in a word graph was conducted using the extended dynamic programming technique proposed by Nagata (1994). This algorithm was originally developed for a statistical Japanese morphological analyzer whose tasks are to determine boundaries in an input character sequence having no separators and to give an appropriate part of speech tag to each word, i.e., a character sequence unit. This algorithm can handle arbitrary lengths of histories of pos tags and words and efficiently produce n-best results.</Paragraph> <Paragraph position="1"> We can see a high similarity between our task and Japanese morphological analysis. Our task requires the segmentation of a word sequence instead of a character sequence and the assignment of an SA tag instead of a pos tag.</Paragraph> <Paragraph position="2"> The main difference is that a word dictionary is available with a morphological analyzer. Thanks to its dictionary, a morphological analyzer can assume possible morpheme boundaries. 4 Our tagger, on the other hand, has to assume that any word sequence in a turn can constitute an SA unit in the search. This difference, however, does not require any essential change in the search algorithm.</Paragraph> </Section> <Section position="7" start_page="383" end_page="385" type="metho"> <SectionTitle> 5 Tagging Experiments </SectionTitle> <Paragraph position="0"/> <Section position="1" start_page="383" end_page="383" type="sub_section"> <SectionTitle> 5.1 Data Profile </SectionTitle> <Paragraph position="0"> We have conducted several tagging experiments on both the Japanese and English corpora described in sub-section 2.1. Table 1 shows a summary of the 95 files used in the experiments. In the experiments described below, we used morpheme sequences for input instead of word sequences and showed the corresponding counts.</Paragraph> <Paragraph position="1"> The average number of SA units per turn was 2.68 for Japanese and 2.31 for English. The average number of boundary candidates per turn was 18 for Japanese and 12.7 for English. The number of tag types, the average number of SA units, and the average number of SA boundary candidates indicated that the Japanese data were more difficult to process.</Paragraph> <Paragraph position="2"> 4Als0, the probability for the existence of a word can be directly estimated from the corpus.</Paragraph> </Section> <Section position="2" start_page="383" end_page="383" type="sub_section"> <SectionTitle> 5.2 Evaluation Methods </SectionTitle> <Paragraph position="0"> We used &quot;labeled bracket matching&quot; for evaluation (Nagata, 1994). The result of tagging can be viewed as a set of labeled brackets, where brackets correspond to turn segmentation and their labels correspond to SA tags. With this in mind, the evaluation was done in the following way. We counted the number of brackets in the correct answer, denoted as R (reference). We also counted the number of brackets in the tagger's output, denoted as S (system). Then the number of matching brackets was counted and denoted as M (match). Thus, we could define the precision rate with M/S and the recall rate with M/R.</Paragraph> <Paragraph position="1"> The matching was judged in two ways. One was &quot;segmentation match&quot;: the positions of both starting and ending brackets (boundaries) were equal.</Paragraph> <Paragraph position="2"> The other was &quot;segmentation+tagging match&quot;: the tags of both brackets were equal in addition to the segmentation match.</Paragraph> <Paragraph position="3"> The proposed evaluation simultaneously confirmed both the starting and ending positions of an SA unit and was more severe than methods that only evaluate one side of the boundary of an SA unit.</Paragraph> <Paragraph position="4"> Notice that the precision and recall for the segmentation+tagging match is bounded by those of the segmentation match.</Paragraph> </Section> <Section position="3" start_page="383" end_page="385" type="sub_section"> <SectionTitle> 5.3 Tagging Results </SectionTitle> <Paragraph position="0"> The total tagging performance is affected by the two probability terms PE and PT, both of which contain the parameters in Table 2. To find the best param- null eter set and see the effect of each parameter, we conducted the following two types of experiments. I Change the parameters for PE with fixed parameters for PT The effect of the parameters in PE was measured by the segmentation match.</Paragraph> <Paragraph position="1"> II Change the parameters for PT with fixed parameters for PE The effect of the parameters in PT was measured by the segmentation+tagging match.</Paragraph> <Paragraph position="2"> Now, we report the details with the Japanese set. We fixed the parameters for PT as f(uj), g(uj), tj-1, i.e., a speaker's identification, cue words in the current SA unit, and the SA tag of the previous SA unit. The unit existence probability was estimated using the following parameters.</Paragraph> <Paragraph position="3"> (A): Surface wordforms and pos's ofw~ +1, i.e., word range r = 1 (B): Surface wordforms and pos's of w x+2 i.e., word x-i, range r ---- 2 (C): (h) with a pause duration between wx, Wx+l (D): (U) with a pause duration between wx, wx+l Under the above conditions, we conducted 10-fold cross-validation tests and measured the average recall and precision rates in the segmentation match, which are listed in Table 3.</Paragraph> <Paragraph position="4"> We then conducted l-tests among these average scores. Table 4 shows the l-scores between different parameter conditions. In the following discussions, we will use the following l-scores: t~=0.0~5(18) -2.10 and t~=0.05(18) = 1.73.</Paragraph> <Paragraph position="5"> We can note the following features from Tables 3 and 4.</Paragraph> <Paragraph position="6"> * recall rate (B), (C), and (D) showed statistically significant (two-sided significance level of 5%, i.e., t > 2.10) improvement from (A). (D) did not show significant improvement from either (B) nor (C).</Paragraph> <Paragraph position="7"> * precision rate Although (n) and (C) did not improve from (A) with a high statistical significance, we can observe the tendency of improvement. (D) did not show a significant difference from (B) or (C).</Paragraph> <Paragraph position="8"> We can, therefore, say that (B) and (C) showed equally significant improvement from (A): expansion of the word range r from I to 2 and using pause information with word range 1. The combination of word range 2 and pause (D), however, did not show any significant differences from (B) or (C). We believe that the combination resulted in data sparseness. For the Type II experiments, we set the parameters for PE as condition (C): surface wordforms and pos's of wx TM and a pause duration between w~ and w~+l. Then, PT was estimated using the following parameters.</Paragraph> <Paragraph position="9"> (E): Cue words in utterance uj, i.e., g(uj) The recall and precision rates for the segmentation/tagging match were evaluated in the same way as in the previous experiments. The results are shown in Table 5. The l-scores among these parameter setting are shown in Table 6. We can observe the following features.</Paragraph> <Paragraph position="10"> * recall rate (F) and (G) showed an improvement from (E) with a two-sided significance level of 10% (1 > Same as recall rate.</Paragraph> <Paragraph position="11"> Here, we can say that tj-1 together with the cue words (F) played the dominant role in the SA tag assignment, and the further addition of history tj-2 (G) or the speaker's identification f(uj) (H) did not result in significant improvements.</Paragraph> </Section> <Section position="4" start_page="385" end_page="385" type="sub_section"> <SectionTitle> 5.3.3 Summary of Japanese Tagging Experiments </SectionTitle> <Paragraph position="0"> As a concise summary, the best recall and precision rates for the segmentation match were obtained with conditions (n) and (C): approximately 92% and 93%, respectively. The best recall and precision rates for the segmentation+tagging match were 74.91% and 75.35 %, respectively (Table 5 (F)). We consider these figures quite satisfactory considering the severeness of our evaluation scheme.</Paragraph> <Paragraph position="1"> We will briefly discuss the experiments with English data. The English corpus experiments were similar to the Japanese ones. For the SA unit segmentation, we changed the word range r from 1 to 3 while fixing the parameters for PT to (H), where we obtained the best results with word range r --- 2, i.e., (B). The recall rate was 71.92% and the precision rate was 78.10%. 5 We conducted the exact same tagging experiments as the Japanese ones by fixing the parameter for PE to (B). Experiments with condition (H) showed the best score: the recall rate was 53.17% and the precision rate was 57.75%. We obtained lower performance than that for Japanese. This was somewhat surprising since we thought English would be easier to process. The lower performance in segmentation affected the total tagging performance.</Paragraph> <Paragraph position="2"> We will further discuss the difference in section 7.</Paragraph> </Section> </Section> <Section position="8" start_page="385" end_page="386" type="metho"> <SectionTitle> 6 Application of SA tags to speech </SectionTitle> <Paragraph position="0"> translation In this section, we will briefly discuss an application of SA tags to a machine translation task. This is one ~Experiments with pause information were not conducted.</Paragraph> <Paragraph position="1"> of the motivations of the automatic tagging research described in the previous sections. We actually dealt with the translation problem of positive responses appearing in both Japanese and English dialogues. Japanese positive responses like Hat and Soudesuka, and the English ones like Yes and I see appear quite often in our corpus. Since our dialogues were collected from the travel arrangement domain, which can basically be viewed as a sequence of a pair of questions and answers, they naturally contain many of these expressions.</Paragraph> <Paragraph position="2"> These expressions are highly ambiguous in wordsense. For example, Hai can mean Yes (accept), Uh huh (acknowledgment), hello (greeting) and so on. Incorrect translation of the expression could confuse the dialogue participants. These expressions, however, are short and do not contain enough clues for proper translation in themselves, so some other contextual information is inevitably required.</Paragraph> <Paragraph position="3"> We assume that SA tags can provide such necessary information since we can distinguish the translations by the SA tags in the parentheses in the above examples.</Paragraph> <Paragraph position="4"> We conducted a series of experiments to verify if positive responses can be properly translated using SA tags with other situational information. We assumed that SA tags are properly given to these expressions and used the manually tagged corpus described in Table 1 for the experiments.</Paragraph> <Paragraph position="5"> We collected Japanese positive responses from the SA units in the corpus. After assigning an English translation to each expression, we categorized these expressions into several representative forms. For example, the surface Japanese expression Ee, Kekkou desu was categorized under the representative form Kekkou.</Paragraph> <Paragraph position="6"> We also made such data for English positive responses. The size of the Japanese and English data in representative forms (equivalent to SA unit) is shown in Table 7. Notice that 1,968 out of 5,416 Japanese SA units are positive responses and 1,037 out of 4,675 English SA units are positive responses. The Japanese data contained 16 types of English translations and the English data contained 12 types of Japanese translations in total.</Paragraph> <Paragraph position="7"> We examined the effects of all possible combinations of the following four features on translation accuracy. We trained decision trees with the C4.5 (Quinlan, 1993) type algorithm while using these features (in all possible combinations) as attributes. null (I) Representative form of the positive response (J) SA tag for the positive response (K) SA tag for the SA unit previous to the positive response (L) Speaker (Hotel/Clerk) We will show some of the results. Table 8 shows the accuracy when using one feature as the attribute. We can naturally assume that the use of feature (I) gives the baseline accuracy.</Paragraph> <Paragraph position="8"> The result gives us a strange impression in that the SA tags for the previous SA units (K) were far more effective than the SA tags for the positive responses themselves (J). This phenomenon can be explained by the variety of tag types given to the utterances. A positive response expressions of the same representative form have at most a few SA tag types, say two, whereas the previous SA units can have many SA tag types. If a positive response expression possesses five translations, they cannot be translated with two SA tags.</Paragraph> <Paragraph position="9"> Table 9 shows the best feature combinations at each number of features from 1 to 4. The best feature combinations were exactly the same for both translation directions, Japanese to English and vice versa. The percentages are the average accuracy obtained by the 10-fold cross-validation, and the t-score in each row indicates the effect of adding one feature from the upper row. We again admit a t-score that is greater than 2.01 as significant (twosided significance level of 5 %).</Paragraph> <Paragraph position="10"> The accuracy for Japanese translation was saturated with the two features (K) and (I). Further addition of any feature did not show any significant improvement. The SA tag for the positive responses did not work.</Paragraph> <Paragraph position="11"> The accuracy for English translation was satu- null rated with the three features (K), (I), and (L). The speaker's identification proved to be effective, unlike Japanese. This is due to the necessity of controlling politeness in Japanese translations according to the speaker. The SA tag for the positive responses did not work either.</Paragraph> <Paragraph position="12"> These results suggest that the SA tag information for the previous SA unit and the speaker's information should be kept in addition to representative forms when we implement the positive response translation system together with the SA tagging system. null</Paragraph> </Section> <Section position="9" start_page="386" end_page="387" type="metho"> <SectionTitle> 7 Related Works and Discussions </SectionTitle> <Paragraph position="0"> We discuss the tagging work in this section. In sub-section 5.3, we showed that Japanese segmentation into SA units was quite successful only with lexical information, but English segmentation was not that successful.</Paragraph> <Paragraph position="1"> Although we do not know of any experiments directly comparable to ours, a recent work reported by Cettolo and Falavigna (1998) seems to be similar. In that paper, they worked on finding semantic boundaries in Italian dialogues with the &quot;appointment scheduling task.&quot; Their semantic boundary nearly corresponds to our SA unit boundary. Cettolo and Falavigna (1998) reported recall and precision rates of 62.8% and 71.8%, respectively, which were obtained with insertion and deletion of boundary markers. These scores are clearly lower than our results with a Japanese segmentation match.</Paragraph> <Paragraph position="2"> Although we should not jump to a generalization, we are tempted to say the Japanese dialogues are easier to segment than western languages. With this in mind, we would like to discuss our study.</Paragraph> <Paragraph position="3"> First of all, was the manual segmentation quality the same for both corpora? As we explained in sub-section 2.1, both corpora were tagged by experts, and the entire result was checked by one of them for each language. Therefore, we believe that there was not such a significant gap in quality that could explain the segmentation performance.</Paragraph> <Paragraph position="4"> Secondly, which lexical information yielded such a performance gap? We investigated the effects of part-of-speech and morphemes in the segmentation of both languages. We conducted the same 10-fold cross-validation tests as in sub-section 5.3 and obtained 82.29% (recall) and 86.16% (precision) for Japanese under condition (B'), which used only pos's in &quot; x+~ for the PE calculation. English, in con- Wx-1 trast, marked rates of 65.63% (recall) and 73.35% (precision) under the same condition. These results indicated the outstanding effectiveness of Japanese pos's in segmentation. Actually, we could see some pos's such as &quot;ending particle (shu-jyoshi)&quot; which clearly indicate sentence endings and we considered that they played important roles in the segmentation. English, on the other hand, did not seem to have such strong segment indicating pos's. Although lexical information is important in English segmentation (Stoleke and Shriberg, 1996), what other information can help improve such segmentation? Hirschberg and Nakatani (1996) showed that prosodic information helps human discourse segmentation. Litman and Passonneau (1995) addressed the usefulness of a &quot;multiple knowledge source&quot; in human and automatic discourse segmentation.</Paragraph> <Paragraph position="5"> Vendittiand Swerts (1996) stated that the intonational features for many Indo-European languages help cue the structure of spoken discourse. Cettolo and Falavigna (1998) reported improvements in Italian semantic boundary detection with acoustic information. All of these works indicate that the use of acoustic or prosodic information is useful, so this is surely one of our future directions. The use of higher syntacticM information is also one of our directions. The SA unit should be a meaningful syntactic unit, although its degree of meaningfulness may be less than that in written texts. The goodness of this aspect can be easily incorporated in our probability term PE.</Paragraph> </Section> class="xml-element"></Paper>