File Information
File: 05-lr/acl_arc_1_sum/cleansed_text/xml_by_section/metho/03/w03-1109_metho.xml
Size: 38,585 bytes
Last Modified: 2025-10-06 14:08:33
<?xml version="1.0" standalone="yes"?> <Paper uid="W03-1109"> <Title>BRIDJE over a Language Barrier: Cross-Language Information Access by Integrating Translation and Retrieval</Title> <Section position="3" start_page="0" end_page="0" type="metho"> <SectionTitle> BRIDJE (Bi-directional Retriever/Information Dis- </SectionTitle> <Paragraph position="0"> tiller for Japanese and English) system (Sakai et al., 2002a; Sakai et al., 2002b; Sakai et al., 2003) which integrates machine translation (MT) with information retrieval (IR) to support both English-Japanese (E-J) and Japanese-English (J-E) CLIA. The first feature is the partial disambiguation function of the Bi-directional Retriever part of BRIDJE for enhancing retrieval performance in the traditional sense. While most of the traditional MT-based CLIR systems use MT as a &quot;black box&quot;, partial disambiguation accesses the internal data structures of a commercial MT system for search request translation so that multiple translation candidates can be used as search terms.</Paragraph> <Paragraph position="1"> We present positive results that are consistent across five test collections (or six topic sets), and across two language permutations: English-Japanese and Japanese-English. To our knowledge, BRIDJE is the first system that truly integrates MT with IR and performs well in terms of standard measures. The second new feature is the entire Information Distiller part of BRIDJE, which can provide generic or query-specific summaries of the retrieved documents, as well as their translations in source language. Based on Semantic Role Analysis (SRA) originally designed for enhancing retrieval performance (Sakai et al., 2002a; Suzuki et al., 2001), the Information Distiller extracts important text fragments from a retrieved document on the fly. Preliminary evaluations suggest that SRA can classify text fragments with very high precision, and that it is useful for efficient information access. We regard our Information Distiller feature as one step towards Cross-Language Question Answering.</Paragraph> <Paragraph position="2"> BRIDJE is an enhanced, fully bilingual version of the KIDS Japanese retrieval system that recently achieved the highest performances in the English-Japanese and Japanese monolingual IR tasks at NTCIR-3 (Chen et al., 2003; Sakai et al., 2003).</Paragraph> <Paragraph position="3"> The remainder of this paper is organised as follows: Section 2 compares some of the previous work on CLIR/CLIA with our present study. Section 3 provides an overview of the BRIDJE system. Section 4 describes an extensive set of retrieval experiments that compares partial disambiguation with the black-box MT approach. Section 5 provides examples to illustrate the advantages of the Information Distiller for efficient information access, as well as some evaluation results of SRA. Finally, Section 6 provides conclusions.</Paragraph> </Section> <Section position="4" start_page="0" end_page="0" type="metho"> <SectionTitle> 2 Previous Work </SectionTitle> <Paragraph position="0"> This section reviews some previous work on CLIR/CLIA, focussing primarily on those that deal with English and Japanese or those that use MT in some way or other.</Paragraph> <Paragraph position="1"> The early J-E CLIR systems (Susaki et al., 1996; Yamabana et al., 1998) employed dictionary-based search request translation, with corpus-based disambiguation. However, it is not clear how effective these systems are in terms of retrieval performance. In contrast, for both J-E and E-J CLIR, MT-based search request translation has been combined successfully with Pseudo-Relevance Feedback (Sakai et al., 1999a; Sakai, 2001). The recent CLIR results at NTCIR-3 also showed that this approach, which BRIDJE also employs, is very promising (Chen et al., 2003; Sakai et al., 2003).</Paragraph> <Paragraph position="2"> In our partial disambiguation experiments, we go beyond the use of MT as a black box by accessing the internal data structures of a commercial MT system in order to use multiple translation candidates as search terms. Jones et al. (Jones et al., 1999) have explored this approach to some extent, but they observed performance degradation when compared to full disambiguation (i.e. black-box MT), as they treated the multiple candidates as distinct search terms. In contrast, BRIDJE treats these candidates as a group of synonyms, which is now known to be effective in CLIR (Pirkola, 1998).</Paragraph> <Paragraph position="3"> Thus, while dictionary-based approaches start from the maximum ambiguity state and performs vigorous disambiguation, we start from the minimum ambiguity state reached through full MT and take a step backward for obtaining alternative translations. The SYSTRAN NLP browser (Gachot et al., 1998) also integrates MT with IR at a deep level, but this was designed for retrieving sentences that match specific grammatical features, and its effectiveness as a document retrieval system is not clear.</Paragraph> <Paragraph position="4"> Regarding CLIA (as opposed to CLIR), Frederking et al. (Frederking et al., 1997) proposed a framework in which retrieved documents could be summarised and translated by multiple MT engines in parallel. MULINEX (Capstick et al., 2000), which is a dictionary-based CLIR system for French, English and German, uses the LOGOS MT system for document/summary presentation. Oard and Rensik (Oard and Resnik, 1999) have used word-by-word J-E translations in their study on interactive document selection. PRIME (Higuchi et al., 2001), yet another J-E/E-J CLIR system based on dictionaries and corpora, translates retrieved patent documents phrase-by-phrase. Compared to these CLIA systems, the Information Distiller part of BRIDJE is unique in that it performs SRA for interactive document summarisation/presentation.</Paragraph> </Section> <Section position="5" start_page="0" end_page="0" type="metho"> <SectionTitle> 3 The BRIDJE System </SectionTitle> <Paragraph position="0"> This section describes the general features of the BRIDJE system. Sections 3.1 and 3.2 describe the request translation and indexing/retrieval components of the Bi-directional Retriever. Section 3.3 introduces the Information Distiller that provides summaries and translations of retrieved documents.</Paragraph> <Section position="1" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 3.1 Request Translation </SectionTitle> <Paragraph position="0"> The default search request translation strategy of BRIDJE is full disambiguation: a source language request is fed to a commercial MT system (Amano et al., 1989), and the output is treated as a monolingual request written in target language. BRIDJE is also capable of performing transliteration for treating words that are outside of the MT dictionary (Sakai et al., 2002b), but this feature is not used in the present study.</Paragraph> <Paragraph position="1"> In order to describe partial disambiguation,we first illustrate the disambiguation process of MT with an example. Suppose that a search request contains the word &quot;play&quot; in the context of E-J CLIR. Firstly, by dictionary lookup, we recognise that &quot;play&quot; can either be a noun or a verb, and obtain all possible Japanese translations accordingly. Secondly, we determine its part-of-speech through syntactic analysis. Here, suppose that &quot;play&quot; was used as a verb, and that all Japanese translations for the noun &quot;play&quot; have been filtered out. Thirdly, we perform semantic analysis using transfer rules. These rules contain knowledge such as &quot;IF the object of the verb &quot;play&quot; is a sport, THEN it should be translated as &quot;suru&quot;. IF the object is a musical instrument, THEN it should be translated as &quot;hiku&quot;or&quot;ens-o-suru&quot;. Here, suppose that the object was &quot;violin&quot;, and therefore that &quot;hiku&quot; and &quot;ens-o-suru&quot; have been selected as translation candidates. Then, at the final output stage, only one translation is selected for each source language word. For the word &quot;play&quot; in the above example, &quot;hiku&quot; is selected as the final translation since this is the first entry in the aforementioned transfer rule.</Paragraph> <Paragraph position="2"> (The above explanation is a simplified version of what really goes on inside our MT system.) Partial disambiguation takes all candidate translations that are left just after the semantic analysis stage (&quot;hiku&quot; and &quot;ens-o-suru&quot; in the above example), and treats them as a set of synonyms in retrieval. As it is well known, disambiguation in MT is far from perfect, and the synonym groups thus produced often contain some inappropriate terms. Despite this, our experiments described in Section 4 show that partial disambiguation is effective.</Paragraph> </Section> <Section position="2" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 3.2 Indexing and Retrieval </SectionTitle> <Paragraph position="0"> Our default retrieval strategy is Okapi/BM25 with Pseudo-Relevance Feedback (PRF) (Robertson and Sparck Jones, 1997; Sakai, 2001). The term selection criterion used in all of our experiments involving PRF is ow4, which incorporates the initial document scores into the traditional offer weight (Sakai et al., 2003).</Paragraph> <Paragraph position="1"> BRIDJE employs word-based indexing, prior to which synonyms and phrases can be defined. Synonym groups can also be defined at query time, which is useful for partial disambiguation and transliteration (Sakai et al., 2002b): the term frequency (tf ) and the document frequency (df ) are counted as if all the members of a synonym group are one identical term.</Paragraph> <Paragraph position="2"> Optionally, BRIDJE can perform indexing and retrieval based on SRA (Sakai et al., 2002a; Suzuki et al., 2001), which was originally designed for going beyond the &quot;bag-of-words&quot; approach to IR. Although the present study uses SRA for interactive document summarisation/presentation and not for retrieval, it works as follows: During document indexing and search request processing, BRIDJE can examine the output of the parser (morphological analyser for Japanese; part-of-speech tagger/stemmer for English), based on a set of hand-written SRA rules which specify the following: How to break up the text into fragments, based on regular expression pattern matching. A fragment can be a sentence, a paragraph, a prepositional phrase, and so on. For example, an English sentence of the form &quot;A for B with C&quot; can be broken into &quot;A&quot;, &quot;for B&quot; and &quot;with C&quot; if prepositions are used as fragment boundaries.</Paragraph> <Paragraph position="3"> How to assign Semantic Roles (SRs) to each fragment, again based on regular expression pattern matching. Using prepositions as SRA triggers in the above example, it is possible to tag the fragment &quot;for B&quot; with &quot;PURPOSE&quot;, the fragment &quot;with C&quot; with &quot;MEANS&quot;, and the fragment &quot;A&quot; with &quot;UNDETERMINED&quot; (which means that no SRA rule matched).</Paragraph> <Paragraph position="4"> The SR correlation weights for document score calculation in retrieval (not used for document presentation). For example, if a term occurs in a PURPOSE context in the request and occurs in a PURPOSE context in the document as well, its BM25 term weight can be upweighted.</Paragraph> <Paragraph position="5"> The above simple example of using PURPOSE and MEANS as SRs has been shown to be effective for retrieving highly relevant documents from the NTCIR-2 English test collection (Sakai et al., 2002a). However, we have also found that it is difficult to devise SRA rules that work across different relevance levels or across different document types.</Paragraph> <Paragraph position="6"> This lead us to explore an alternative way of utilising SRA, i.e. for document summarisation/presentation.</Paragraph> </Section> <Section position="3" start_page="0" end_page="0" type="sub_section"> <SectionTitle> 3.3 Information Distiller </SectionTitle> <Paragraph position="0"> The Information Distiller can interactively generate generic or query-specific summaries of a retrieved document by extracting fragments (usually sentences) that meet specified criteria. The unique feature of BRIDJE as a CLIA system is that it supports interactive selection of fragments based on SRA: For example, the user can select fragments whose SRs are TOPIC/AIM, BACKGROUND, RESULT/CONCLUSION, OPINION, and so on. In addition, the Information Distiller can generate lead and tf -idf -based extracts, and these criteria can be combined with SRA requirements. The summaries thus produced (or the original document text) can be translated back into source language using MT.</Paragraph> </Section> </Section> <Section position="6" start_page="0" end_page="1" type="metho"> <SectionTitle> 4 Evaluation of Partial Disambiguation </SectionTitle> <Paragraph position="0"> This section compares partial disambiguation with traditional full disambiguation for search request translation in CLIR.</Paragraph> <Section position="1" start_page="0" end_page="1" type="sub_section"> <SectionTitle> 4.1 Experimental Setting </SectionTitle> <Paragraph position="0"> We used three English-Japanese test collections:</Paragraph> <Paragraph position="2"> has two English topic sets X and Y translated from a single Japanese topic set (Sakai et al., 1999a), we Data in BMIR-J2 is taken from the Mainichi Shimbun CD-ROM 1994 data collection. BMIR-J2 was constructed by the SIG Database Systems of the Information Processing Society of Japan, in collaboration with the Real World Computing Partnership. null performed four sets of E-J CLIR experiments in total. On the other hand, we used two Japanese-English test collections: NTCIR-3 J-E (Chen et al., 2003) and NTCIR-2 J-E (Kando, 2001). Table 1 summarises the features of these five test collections. As they provide multiple relevance levels, the number of relevant documents summed across topics are shown for each relevance level: S-relevant (highly relevant), A-relevant (relevant), and B-relevant (partially relevant). There are no S-relevant documents for BMIR-J2.</Paragraph> <Paragraph position="3"> Following the practice at NTCIR, we computed Mean Average Precision (MAP) based on &quot;relaxed&quot; relevance (which treats S,A and B-relevant documents as relevant) and on &quot;rigid&quot; relevance (which treats S and A-relevant documents as relevant) (Chen et al., 2003; Kando, 2001). In addition, for unified evaluation with multiple relevance levels, we used Average Gain Ratio (AGR) (Sakai et al., 2003; Sakai, 2003). For testing statistical significance, we used the sign test.</Paragraph> <Paragraph position="4"> For each test collection, full disambiguation and partial disambiguation queries were generated using the topic descriptions. Although our MT system has several domain-specific dictionaries, we used the general dictionary only. For the NTCIR-3 E-J experiment, the BM25 and PRF parameters were tuned using the dryrun topics (Sakai et al., 2003). For all other experiments, Okapi/BM25 defaults were used, and the number of pseudo-relevant documents and that of expansion terms were fixed to 10 and 30, respectively (Sakai, 2001).</Paragraph> </Section> </Section> <Section position="7" start_page="1" end_page="1" type="metho"> <SectionTitle> 4.2 Results </SectionTitle> <Paragraph position="0"> Table 2 summarises the results of our CLIR experiments. Full disambiguation runs with and without PRF are denoted by FD+PRF and FD, while the corresponding partial disambiguation runs are denoted by PD+PRF and PD, respectively. The monolingual performances with and without PRF, denoted by ML+PRF and ML, are also shown. Columns (i), (ii) and (iii) show performances in MAP with relaxed relevance, MAP with rigid relevance, and Mean AGR (MAGR), respectively. Runs that significantly outperform FD are indicated by &quot;&quot;s, while those that significantly outperform PD are indicated by &quot;y&quot;s. For example, Table 2A Column (i) include the following information in terms of relaxed MAP: (a) PD+PRF is 8% better than FD+PRF, and significantly better than FD ( =0:01) and PD ( =0:05); (b) FD+PRF is not significantly better than FD and PD (hence the lack of &quot;&quot;s and &quot;y&quot;s); and (c) PD is 6% better than FD, but this difference is not statistically significant (hence the lack of &quot;&quot;s). The following are general observations made from</Paragraph> <Paragraph position="2"> terms of all three evaluation measures. The improvements are 1-11%. The differences are statistically significant in Table 2C Columns (i) and (iii), for Topic Set Y ( =0:05).</Paragraph> <Paragraph position="3"> PD+PRF outperforms FD+PRF for all test collections in terms of all three evaluation measures. The improvements are 1-8%. Although these differences are not statistically significant by direct comparison, the &quot;&quot;s and &quot;y&quot;s, which indicate superiority over FD and PD, suggest that PD+PRF is superior to FD+PRF in general. (Table 2B is an exception: here, there is no statistical evidence which suggests that PD+PRF is superior to FD+PRF. However, note that PD+PRF outperforms FD+PRF on average even for this test collection.) In general, PRF preserves the positive effect of partial disambiguation. For example, Table 2A Column (ii) shows that PD is 8% better than FD, and that PD+PRF is also 8% better than FD+PRF. (Table 2C is an exception: For example, in Column (ii), PD+PRF(Y ) is only 2% better than FD+PRF(Y ) even though PD(Y )is 11% better than FD(Y ).) Thus, the small advantage of partial disambiguation over full disambiguation is consistent across the five test collections (six topic sets) and across the two language permutations.</Paragraph> <Paragraph position="4"> Table 3(a) compares FD and PD for each test collection in terms of average number of query terms per topic. For the E-J topics, the PD queries are approximately three times longer than the FD ones. Compared to this, the FD and PD queries for the J-E topics are relatively similar in length. That is, semantic analysis in J-E MT generally yields fewer translation candidates than that in E-J MT does. However, the relationship between the number of alternative translations and the success of partial disambiguation is not straightforward: While the E-J results are more successful than the J-E results for NTCIR-3 (Table 2A vs D), suggesting that &quot;adding more terms is better&quot;, this is not true for NTCIR-2 (Table 2B vs E). This is probably because the quality of the alternative translations vary widely, as we shall see later.</Paragraph> <Paragraph position="5"> Table 3(b) shows the total number of out-of-vocabulary words for each topic set. Neither FD nor PD could translate these words as our general MT dictionary was not tuned in any way for our experiments. The NTCIR-3 E-J topic set contained three out-of-vocabulary words, including &quot;Tomiich&quot; which is a misspelling of &quot;Tomiichi (Murayama, a former Japanese prime minister)&quot;, while the NTCIR-3 J-E topic set contains five, including &quot;konpyut-a&quot; which is a misspelling of &quot;konpy-ut-a&quot;or &quot;konpy-uta&quot;. Meanwhile, almost all of the out-of-vocabulary words in the NTCIR-2 E-J and J-E topic sets were technical terms. However, there was no topic that contained more than one out-of-vocabulary Runs that significantly outperform FD are indicated by &quot;&quot;( =0:05) and &quot;&quot;( =0:01). Those that significantly outperform PD are indicated by &quot;y&quot;( =0:05) and &quot;yy&quot;( =0:01). The percentages in the PD+PRF rows represent the gain over FD+PRF, while those in the PD rows represent the gain over FD. word (with one exception), and it appears that out-of-vocabulary words did not directly affect our experiments: partial disambiguation managed to improve five of the eight NTCIR-2 J-E topics that contained an out-of-vocabulary word.</Paragraph> <Paragraph position="6"> Given that the effect of out-of-vocabulary words is negligible, one may hypothesize that the advantage of partial disambiguation over full disambiguation may be smaller with technical papers than with newspapers, as technical papers contain more technical terms and full disambiguation may be sufficient for translating them. By comparing the E-J results (Table 2A-C), it can be observed that partial disambiguation was indeed a little less successful for NTCIR-2 E-J (technical papers) than for NTCIR-</Paragraph> </Section> <Section position="8" start_page="1" end_page="1" type="metho"> <SectionTitle> 3 E-J and BMIR-J2 E-J (newspapers). However, </SectionTitle> <Paragraph position="0"> as our J-E results (Table 2D-E) show comparable performances for both document types, the above hypothesis is not fully supported.</Paragraph> <Section position="1" start_page="1" end_page="1" type="sub_section"> <SectionTitle> 4.3 Per-topic Analyses </SectionTitle> <Paragraph position="0"> While partial disambiguation improves retrieval performance on average for all test collections, per-topic analyses show that it hurts performance for some topics, and that there is room for improvement. Table 4 provides some per-topic comparisons of the Average Precision values of FD and PD. Two examples are given for NTCIR-3 E-J and for J-E, respectively, where the Japanese descriptions for the latter are shown here in English. Words that are mentioned in the discussion below are underlined.</Paragraph> <Paragraph position="1"> As Table 4 shows, PD was hugely successful for E-J Topic 017: The only word that FD translated correctly was &quot;Kitano&quot;: &quot;Director&quot; was mistranslated as &quot;kanrisha&quot; (manager), &quot;Takeshi&quot; was transliterated into katakana (which is not appropriate in this case), and &quot;films&quot; was transliterated into &quot;firumu&quot; (which means &quot;camera films&quot;, not &quot;movies&quot;). In contrast, PD successfully recaptured the correct translations &quot;kantoku&quot; (director) and &quot;eiga&quot; (films). However, some inappropriate translations such as &quot;shikisha&quot; (orchestra director) and &quot;torishimariyaku&quot; (managing director) were added as well, as semantic analysis is not perfect. Moreover, PD did not help in translating &quot;Takeshi&quot;: although it obtained two kanji spellings for it, they were incorrect for this particular Takeshi Kitano. (There are more than 40 possible kanji spellings for &quot;Takeshi&quot;!) Nevertheless, the use of synonym operators seems to have absorbed the negative effect of such inappropriate translations for this topic. On the other hand, PD was not successful for E-J Topic 004: while FD obtained the correct translations &quot;denshish-otorihiki&quot; (E-commerce) and &quot;naiy-o&quot; (contents), PD added the acronym of &quot;E-commerce&quot;, which happened to hit many nonrelevant documents that mention &quot;European Community&quot;. (The roman alphabet is often used in Japanese texts for representing foreign acronyms.) Moreover, PD added inappropriate translations for &quot;contents&quot; such as &quot;mokuji&quot; (table of contents) and &quot;y-ory-o&quot; (capacity). null PD was also successful for J-E Topic 031: FD obtained &quot;optimal&quot; instead of &quot;best&quot;, and &quot;place&quot; instead of &quot;spot&quot;. In contrast, PD successfully recaptured &quot;best&quot; and &quot;spot&quot;, even though it also added some possibly harmful terms such as &quot;position&quot; and &quot;space&quot;. On the other hand, PD was not successful for J-E Topic 050: FD obtained &quot;dress&quot; instead of &quot;clothing&quot;, to which PD added &quot;appearance&quot;, &quot;clothes&quot;, &quot;costume&quot; and &quot;garment&quot;. Meanwhile, FD obtained &quot;hairstyle&quot; instead of &quot;hair styles&quot;, to which PD added &quot;hairdo&quot; and &quot;coiffure&quot;. FD obtained &quot;makeup&quot; instead of &quot;cosmetics&quot;, to which PD added &quot;dressing&quot; and &quot;toilet&quot;.</Paragraph> <Paragraph position="2"> As mentioned earlier, while semantic analysis in J-E MT generally yields fewer translations than that in E-J MT does, this difference is not clearly reflected in terms of retrieval performance. One possible cause of this is that the partial disambiguation terms in the J-E case are more polysemous, e.g. &quot;space&quot; and &quot;toilet&quot;, though fewer in number.</Paragraph> <Paragraph position="3"> The above examples suggest that partial disambiguation may be improved by adopting a more selective strategy. Although we have conducted additional experiments by limiting the number of terms added by partial disambiguation, this did not improve performance as the candidate terms obtained after semantic analysis have no priority information in our MT system. One possible solution to this is to utilise the corpus statistics such as the document frequency so that polysemous words can be filtered out, but this is beyond the scope of this paper.</Paragraph> <Paragraph position="4"> Finally, by looking across the columns of Table 2, it can be observed that the results in terms of MAGR are generally consistent with those in terms of re- null S1: It is generally thought that the increase in blood pressure with age may be avoided by the extremely low sodium intake. (BACKGROUND) S2: we, however, have noticed that blood pressure hardly increased with age in some communities briefly studied in Nepal since 1978, even though the inhabitants take salty foods and beverages.(TOPIC/AIM) S3: Our purpose was to ascertain this and to clarify the factor(s) influencing no increase in blood pressure with age in terms of extensive epidemiological standpoint. (TOPIC/AIM) S9: RESULTS : blood pressure for both sexes was statistically significantly higher in villagers in Bhadrakali than in Kotyang. (RESULT/CONCLUSION) S16: CONCLUSION : In spite of consuming more than 10g per day of salt in both Kotyang and Bhadrakali, the blood pressure hardly increased with age only in the former, suggesting that the blood pressure may be influenced by physical activity, fat free mass and nutrient consumption rather than salt intake in these villages in Nepal. (RESULT/CONCLUSION)</Paragraph> </Section> </Section> <Section position="9" start_page="1" end_page="1" type="metho"> <SectionTitle> 5 Information Distiller </SectionTitle> <Paragraph position="0"> This section provides some example summaries and translations to illustrate the usefulness of the Information Ditiller for efficient information access, as well as some preliminary evaluation results of SRA.</Paragraph> <Section position="1" start_page="1" end_page="1" type="sub_section"> <SectionTitle> 5.1 Example Summaries and Translations </SectionTitle> <Paragraph position="0"> As this paper is intended primarily for the English speaking community, we provide one example summary in the context of J-E CLIR before they are translated by MT into Japanese, and two in the context of E-J CLIR after they have been translated by MT into English. The summaries shown here are all query-specific, and are based on full disambiguation queries.</Paragraph> <Paragraph position="1"> Figure 1 shows a sample summary of an English technical paper abstract that is S-relevant to NTCIR-2 J-E Topic 0132, which is about &quot;correlations between hypertension and diet.&quot; Of the 16 sentences in the original document (excluding the title, shown at the top of this summary), those which do not contain any of the English query terms and those whose SRs were UNDETERMINED have been filtered out, leaving only five sentences. The words that matched the query terms are shown in boldface, and the English expressions which acted as SRA triggers are underlined. For example, Sentence 1 (S1) was tagged with BACKGROUND because the string &quot;It is generally thought&quot; matched a regular expression in one of the SRA rules. If the user desires an indicative summary, BRIDJE can present S2 and S3, which are tagged withTOPIC/AIM. Subsequently, if he desires an informative summary, BRIDJE can present S9 and S16, which are tagged with RESULT/CONCLUSION.Of course, the user can specify multiple SRs, choose to read UNDETERMINED sentences, or combine such usage with tf -idf -based sentence filtering. Real-time response is easy because SRA for the documents are done at index time. Finally, the summary can be translated into Japanese by MT for the Japanese user.</Paragraph> <Paragraph position="2"> In the above example, only one SR was assigned The Cannes International Film Festival is challenged shortly. - Director Takeshi Kitano and &quot;chrysanthemum Jiro's summer&quot; are sent. (TITLE) to each fragment (i.e. sentence) for simplicity. However, SRs can be used as orthogonal features, as shown in the next example.</Paragraph> <Paragraph position="3"> Figure 2 shows a sample translated summary (i.e.</Paragraph> <Paragraph position="4"> an MT output) of a Japanese newspaper article that is S-relevant to NTCIR-3 E-J Topic 017: &quot;Articles relating [related] to Director Takeshi Kitano's films.&quot; Words that match those from the source language request are indicated in boldface, and words that correspond to the Japanese SRA triggers are underlined. As the search request ends with the word &quot;films&quot;, it is possible for BRIDJE to guess that movie titles may be useful to the user. Thus, BRIDJE can show S1 and S4 to the user by default, as they have been tagged with TITLE based on SRA triggers such as &quot;kantoku&quot; (director), &quot;sakuhin&quot; (work) and Japanese brackets. Note also that S1 has two more SRs, TOPIC and DATE. Unfortunately, the correct English translation of the movie title is &quot;Kikujiro's summer&quot;: Kikujiro is a very unusual Japanese name, while Jiro is a common first name. Besides, kiku does mean chrysanthemum the flower! Nevertheless, we view this example as one step towards cross-language question answering, which deals with questions such as &quot;List up Takeshi Kitano's Japanese films - give me rough translations in English.&quot; Similarly, Figure 3 shows a translated summary of a Japanese newspaper article that is S-relevant to NTCIR-3 E-J Topic 030: &quot;When, if ever, has the Japanese Emperor been to Denmark?&quot;. By performing SRA on this &quot;when&quot; question, it is possible for BRIDJE to guess that the user is looking for DATEtype information, as in question answering. In such a case, BRIDJE can present a list of sentences containing dates as shown. Even though the translations are far from perfect, the English speaking user can probably guess, by reading S3, S5, and S6 that the answer to the question is &quot;from May 31st to June 4th or 5th&quot;. Combining this with the meta-data of this document, a sophisticated cross-language question answering system would output the year &quot;1998&quot; as well. Unfortunately, the document title in Figure 3 contains a kanji word which MT failed to translate: &quot;h-o-obi&quot; (dates for visiting Europe), shown as &quot;&quot;in this paper to avoid Japanese fonts.</Paragraph> <Paragraph position="5"> The above three examples illustrate the usefulness of SRA for efficient access to the desired information within an English or a Japanese document, and for allowing different views for summarising a document. Moreover, we have argued that current MT technology can be useful for CLIA despite its limited quality. By performing SRA-based sentence filtering, the amount of MT output that the user has to go through can be kept to a minimum.</Paragraph> </Section> <Section position="2" start_page="1" end_page="1" type="sub_section"> <SectionTitle> 5.2 Precision of SRA </SectionTitle> <Paragraph position="0"> As the Information Distiller is an interactive subsystem, user-oriented, overall usefulness evaluations should be performed. As a first step, however, we evaluate the precision of the SRA rule sets that were actually used to generate the aforementioned examples. The results reported here are only indicative as the rule sets are experimental versions.</Paragraph> <Paragraph position="1"> By using S-relevant documents as training data, one SRA rule set was devised for each of the four document collections: NTCIR-2 English/Japanese (technical papers) and NTCIR-3 English/Japanese (newspapers). Then, for each collection, we prepared a test set of documents A as follows: Let S and A be the set of unique S-relevant and A-relevant documents, respectively. Take 100 A-relevant documents at random from the set A ; S and let this for each collection. Then, the first author examined each fragment and judged whether the assigned SR was &quot;acceptable&quot; or not. Here, &quot;unacceptable&quot; SRs are those that would probably mislead the user: For example, if a fragment from a technical paper is tagged with RESULT/CONCLUSION even though the fragment in fact provides a BACKGROUND information, this may cause a misunderstanding and is therefore unacceptable. Note also that this evaluation concerns generic summary fragments, as they subsume query-focused ones.</Paragraph> <Paragraph position="2"> We define SRA precision as the number of fragments with acceptable SRs divided by the total number of fragments. We do not consider its recall counterpart here, because the Information Distiller is supposed to filter out many sentences and present &quot;typical&quot; sentences only. Although our current SRA rule sets assign SRs to only a small fraction of given fragments, the user can choose to readUNDETERMINED sentences or select multiple SRs at any time, as discussed earlier. Thus, the aim of this lenient evaluation is to ensure that the output of Information Distiller will &quot;look okay&quot; to the user.</Paragraph> <Paragraph position="3"> Tables 5 and 6 summarise our SRA precision results. Table 5A corresponds to the NTCIR-2 English SRA rule set (which was used to generate the summary in Figure 1), and 5B corresponds to the responds to the NTCIR-3 English SRA rule set, and 6D corresponds to the NTCIR-3 Japanese SRA rule set (which was used to generate the original Japanese summaries for Figures 2 and 3). For example, Table 5A includes information such as: , only 138 (23%) were tagged with an SR; (b) Of the above 138 fragments, 58 were tagged with RESULT/CONCLUSION, but one of them was judged as unacceptable. Hence the Precision for this SR is 57/58=98%; (c) As the abovementioned fragment was the only unacceptable one, the overall precision is 137/138=99%.</Paragraph> <Paragraph position="4"> The one unacceptable fragment tagged with</Paragraph> </Section> </Section> <Section position="10" start_page="1" end_page="1" type="metho"> <SectionTitle> RESULT/CONCLUSION was: &quot;As Kipps's recog- </SectionTitle> <Paragraph position="0"> nition algorithm does not give us a way to extract any parsing result, his algorithm is not considered as a practical parsing algorithm.&quot; which accidentally matched an SRA rule that included &quot;result&quot; as a trigger. As shown in Table 7A, this fragment should probably be tagged withBACKGROUND, since it discusses previous work rather than the author's present work.</Paragraph> <Paragraph position="1"> Similarly, Tables 5B and 7B show the results for the NTCIR-2 Japanese SRA rule set which is very similar to its English counterpart. As there were five unacceptable cases, its overall precision is 97%.</Paragraph> <Paragraph position="2"> Tables 6C and 7C show the results for the NTCIR-</Paragraph> </Section> <Section position="11" start_page="1" end_page="1" type="metho"> <SectionTitle> 3 English SRA rule set. Three fragments tagged with </SectionTitle> <Paragraph position="0"> TOPIC were judged as unacceptable, one of which was: &quot;But I've always said that I won't compromise when it comes to demanding that the facts surrounding the incident come out in the open.&quot; This fragment accidentally matched an SRA trigger &quot;said that&quot;, designed to match fragments such as &quot;The prime minister said that. . . &quot; Tables 6D and 7D show the results for the NTCIR-</Paragraph> </Section> <Section position="12" start_page="1" end_page="1" type="metho"> <SectionTitle> 3 Japanese SRA rule set. Unlike the other SRA rule </SectionTitle> <Paragraph position="0"> sets, the SRs in this rule set were defined as orthogonal features, which allowed multiple SRs per fragment as in Figure 2. (Hence the SubTotal row provides information on distinct fragments.) As many as 18 fragments were incorrectly tagged with COMMENT, due to our reliance on Japanese brackets as SRA triggers: words surrounded by brackets are often technical terms or movie/book titles, rather than quoted comments. Moreover, two fragments were incorrectly tagged withDATE as they contained the word &quot;ichinichi&quot; (one day), whose spelling is the same as &quot;tsuitachi&quot; (first [of January]).</Paragraph> <Paragraph position="1"> To summarise our preliminary results: Our experimental SRA rule sets can assign SRs to 13-31% of completely unknown English/Japanese sentences (but from known document genres), with precision of 95-99%. Moreover, SRA precision would be even higher for query-focused summaries. Thus, the SRs presented by the Information Distiller would probably look satisfactory to the user.</Paragraph> </Section> class="xml-element"></Paper>