<?xml version="1.0" standalone="yes"?>
<Paper uid="W03-1607">
  <Title>Criterion for Judging Request Intention in Response texts of Open-ended Questionnaires</Title>
  <Section position="2" start_page="0" end_page="6" type="intro">
    <SectionTitle>
1 Introduction
</SectionTitle>
    <Paragraph position="0"> In every aspect of society, it is necessary for us to &amp;quot;know what the request is.&amp;quot; This is because knowing what the request is plays an important role in allowing us to identify and solve problems to achieve improvements.</Paragraph>
    <Paragraph position="1"> In recent years, the spread of electronic devices such as personal computers and the Internet has allowed us to save most requests in machine-readable texts. On the basis of these texts, research and development have been conducted &amp;quot;to know what the request is&amp;quot; as an element technology in natural language processing. For example, the research includes text mining (Nasukawa, 2001) and information extraction (Tateno, 2003) for customer claims and inquiries, development of an FAQ generation support system to a call center (Yanase et al., 2002; Matsuzawa, 2002), an FAQ navigation system using Q&amp;A stored a call center (Matsui, 2002), and the development of requirement capturing methods for extracting requests made in meetings for software development (Doi, 2003). However, &amp;quot;to know what the request is&amp;quot; means to know the intention of various people in society such as residents, users, customers and patients, and it is inadequate to extract only request expressions expressed literally in texts. For this reason, previous works are not sufficient to understand intentions.</Paragraph>
    <Paragraph position="2"> Against this background, (Inui et al., 1998; Inui et al., 2001; Inui and Isahara, 2002) have been studying how to extract and classify request intentions of respondents from responses of open-ended questionnaires (OEQs) which are accumulated requests. This paper describes the development of a criterion for judging request intentions and an evaluation of the criterion in terms of objectivity, reproducibility and effectiveness.</Paragraph>
    <Paragraph position="3">  2 Development of the criterion for judging request intentions 2.1 Problems of an existing theory of modality  Response texts of OEQs are the focus of attention as data for text mining. Researchers have tried to extract various types of information from those texts (Lebart et al., 1998; Li and Yamanishi, 2001; Osumi and Lebart, 2000; Takahashi, 2000).</Paragraph>
    <Paragraph position="4"> However, they have mainly used only keywords (mostly nouns) as the basic units of extraction. If only the characteristic key words are analyzed with regard to sentences such as &amp;quot;Company A's beer tastes good,&amp;quot; &amp;quot;Company A's beer does not seem to taste good,&amp;quot; and &amp;quot;Company B's beer tastes better than company A's,&amp;quot; the attention is directed toward &amp;quot;company A/company B/beer/tastes/good,&amp;quot; and it is not possible to differentiate the meaning of the passages.</Paragraph>
    <Paragraph position="5"> Because of this, as (Toyoda, 2002) points out, text mining in the future needs to treat modality, which often changes the meaning of the sentences completely. Two separate studies (Inui et al., 1998; Morohashi et al., 1998) have tried to process texts using words like auxiliary verbs and auxiliary verb equivalents as modality information. The modality information focused on in both studies, however, is grammatical expressions that have been accepted in a previous Japanese language study. Therefore, it is not possible to mechanically interpret requests and questions displayed by respondents, speakers and writers if they don't contain an auxiliary verb or an auxiliary verb equivalent.</Paragraph>
    <Paragraph position="6"> In Japanese language syntax, modality is defined as the intention of the writer that is represented by grammatical expressions expressed grammatically (Nitta and Masuoka ed., 1989) and typically appears in the form of particles and auxiliary verbs in the sentence structure. Although previous text mining has focused on these expressions, modality does not always appear in the forms of grammatical expressions, and other expressions are more frequently used in real world texts. Thus, processing only those grammatical expressions listed so far is not sufficient for extracting intentions, and it is necessary to have a wide coverage of modality that expresses intentions.</Paragraph>
    <Section position="1" start_page="0" end_page="2" type="sub_section">
      <SectionTitle>
2.2 Criterion to judge request intentions
</SectionTitle>
      <Paragraph position="0"> using paraphrasing Surveyors try to know request intentions on the respondents through questionnaires, and respondents try to convey their request intentions to surveyors by responding to questionnaires. Therefore, it is important to establish a method that can extract the request intentions of the respondents based on the expressions given in the response texts. In this section, we propose a criterion to judge the existence of request intentions.</Paragraph>
      <Paragraph position="1"> First, we will analyze the request expressions deductively. Native Japanese speakers can recognize expressions such as te-hoshii (would like you to), te-moraitai (would like you to), te-kudasai (please do) and te-kure (do) as request. These are linguistically called direct request expressions  (NIJLA, 1960) and able to indicate request intentions. Especially, te-hoshii is a typical request expression.</Paragraph>
      <Paragraph position="2"> In other words, these direct request expressions are a clue to understand that there is a request intended. This recognition process is equivalent to the first judgment in Fig.1, that is, &amp;quot;whether a response can be judged to be a request by linguistic intuition or not.&amp;quot; We regarded this as the first level criterion to judge request intentions. It corresponds to the first level in Fig.2, the intent of which is equal to judge whether the response includes a direct request expression or not.</Paragraph>
      <Paragraph position="3"> Second, we consider the case that a response does not contain a direct request expression. In this case, non-requests in Fig.1 may be judged as requests. For example, based on the relation with surveyors, respondents and the situation, &amp;quot;Guardrails should be built along sidewalks of heavily congested roads&amp;quot; and &amp;quot;Building ecofriendly roads is important&amp;quot; can be interpreted as &amp;quot;We want guardrails along the sidewalks&amp;quot; and &amp;quot;We want you to think about the environment.&amp;quot; However, the interpretation is due to &amp;quot;some&amp;quot; implicit criterion as shown in the second judgment in Fig. 1. As the implicit criterion depends on the judges, it is possible that the judgments differ  .</Paragraph>
      <Paragraph position="4"> This means that the results of the judgment, namely request in Fig. 1, are not re-created consistently. Therefore, the second judge in Fig.1 is not reproducible.</Paragraph>
      <Paragraph position="5"> Consequently we attempted to manifest the implicit criterion as an explicit criterion to judge the existence of request intentions. This manifestation is the criterion &amp;quot;whether a response can be paraphrased into a sentence containing te-hoshii as a typical request expression or not&amp;quot; as the second judge in Fig. 2. As this criterion is explicit, the judgment of the criterion does not depend on the judges and agree consistently. Therefore, the second judge in Fig.2, namely the proposed criterion is reproducible and the results of the judgment, namely the expression of request intentions in Fig.2 is re-created consistently  .</Paragraph>
      <Paragraph position="6"> As mentioned above, we propose a criterion for judging request intentions by paraphrasing a response sentence into a typical request sentence  This is demonstrated by the results of the experiment described in Section 4.2.</Paragraph>
      <Paragraph position="7">  This reproducibility is described in detail in Section 4.1. contained te-hoshii. In Section 3, we evaluate the proposed criterion by a single judge analytically and objectively. In Section 4, we evaluate the results of experiments conducted by different judges from the viewpoint of reproducibility and effectiveness. These evaluations enable to demonstrate that the criterion, namely paraphrasing is an important method to determine the intentions independent of variety of surface expressions and differences among individual judgments.</Paragraph>
      <Paragraph position="8"> 3 Evaluation by a single judge</Paragraph>
    </Section>
    <Section position="2" start_page="2" end_page="3" type="sub_section">
      <SectionTitle>
3.1 Analysis of response texts
</SectionTitle>
      <Paragraph position="0"> Using the proposed criterion described in Section 2.2, we analyzed and classified response sentences manually according to two considerations: (1) if they include direct request expressions such as te-hoshii and te-moraitai; and (2) if it is possible to paraphrase them into a sentence ending with tehoshii. To make the judgment for (1), we used request expressions listed by (Morita and Matsui,  The analysis data are part of the response texts of OEQs carried out to make the best use of the opinions of the citizens in future road planning (Voice report, 1996). The original OEQ corpus contains a total of 35,674 respondents and 113,316 opinions. The analysis data comprised 3,000 sentences sampled at random after separating the plural sentences contained in the response text into single sentences. The criterion in Section 2.2 was used and the results are shown in Table 1.</Paragraph>
      <Paragraph position="1"> Line in Table 1 includes sentences with direct request expressions such as te-hoshii, te-kudasai and te-kure. All of these could be paraphrased into te-hoshii and accounted for about 20% of the 3,000 sentences. Line includes direct request expressions that could not be paraphrased because they were used in quotations. These examples are exceptional. Expressions in line correspond to  Eight sentences were excluded from Table 1 because they were ambiguous out of contexts.</Paragraph>
      <Paragraph position="2"> expressions of request intentions in Fig.2 in Section 2.2. These expressions are shown in Table 2. Line includes non-request expressions.</Paragraph>
      <Paragraph position="3"> Table 2 shows various forms of expressions based on parts of speech (POS), i.e., verbs, nouns and adjectives, that have not been considered acceptable as modality expressions, even though they are paraphrasable by te-hoshii, and thus they are request expressions. As described in Section 2.1, several studies have been made on modality in terms of particles, auxiliary verbs, and auxiliary verb equivalents. However, little attention has been given to other POS in this regard. This is because modality expressions have been primarily connected with the grammatical elements such as auxiliary verbs in syntax. However, Table 2, which lists expressions of request intentions, shows that verbs, nouns and adjectives are actually also important elements that express modality.</Paragraph>
      <Paragraph position="4"> Previous works that aim to extract requests have used pattern matching methods, and patterns that mainly consist of the direct request expressions corresponding to in Table 1. However, the results of manual analysis for paraphrasability shown in Table 2 indicate that using the proposed criterion enables many expressions of request intentions to be extracted from responses. In addition, we found a tendency for the number of expressions of request intentions direct request expressions, as shown in Table 1. In this section, we have provided explanation for the coverage of the criterion by analyzing response texts.</Paragraph>
    </Section>
    <Section position="3" start_page="3" end_page="4" type="sub_section">
      <SectionTitle>
3.2 Evaluation of objectivity through
</SectionTitle>
      <Paragraph position="0"> machine learning methods This section shows that the possibility of paraphrasing is learnable by machine learning methods. The data for the machine learning methods were tagged by the expert that analyzed the data in Table 1. Our assumption is that if machine learning methods can learn the paraphrasability from the data, then the data are said to have been tagged consistently enough to be mechanically learnable. This indicates that the criterion proposed in Section 2 is objectively applicable to tag data.</Paragraph>
      <Paragraph position="1"> Machine learning methods We use two machine learning methods in this section. They are maximum entropy method (ME) (Beger et al. 96) and support vector machine</Paragraph>
      <Paragraph position="3"> , both of which have been shown to be quite effective in natural language processing.</Paragraph>
      <Paragraph position="4"> The task of a machine learning method is to make a classifier that can decide whether a response is paraphrasable by te-hoshii or not. A response X is tagged possible if it is paraphrasable  -wo etc. Lao Ren yaZi Gong yaZhang Hai Zhe noLi Chang denoDao dukuriwo . (Road building from the standpoint of the elderly, children, and the disabled)</Paragraph>
    </Section>
    <Section position="4" start_page="4" end_page="6" type="sub_section">
      <SectionTitle>
Verbs and
adjectives of
</SectionTitle>
      <Paragraph position="0"> expectation and desire -woQiu meru (seek) /-niQi Dai suru (expect) / -Yuan i tai (desire) /-gaWang masii (is desirable) /-gaWang mareru (is desired)/-woWang mu (desire) /-woYao Wang su ru (request) etc.</Paragraph>
      <Paragraph position="1"> Zhang Hai Zhe , Lao Ren , Zi Gong , Li Chang noRuo iZhe gaYou Xian siteTong re ruDao gaWang mareru . (Roads and streets that give priority to the disabled, the elderly, children, and the weak are desirable) &lt;attribute: emergency&gt; -gaJi Wu dearu (matter of urgency) /-gaZui You Xian da (first priority) /-gaXian Jue datoSi u (think that the first thing to do) etc.</Paragraph>
      <Paragraph position="2"> Di Fang niGao Su Dao woJian She surunomoiikedo, Shibui Zhi Ge Suo wo Zheng Bei siteikukotogaXian Jue dehanaika. (It is all right to build expressways in provincial areas, but why can't improving congested places come first?) &lt;attribute:importance&gt; -gaZhong Yao da (is important) /-moDa Shi naWen Ti datoSi u (think that it is also an important matter) /-ga Da Qie da (is important) /-gaDa Qie darou (should be important) /-gaLi Xiang (that is ideal) etc.</Paragraph>
      <Paragraph position="3"> Ting Che nomananoChe Di moDa Shi naWen Ti datoSi imasu. (I think that the important matter is to make the manner of stopping vehicles thorough )  using the criterion for judging request intentions and impossible if not. X is represented by a feature</Paragraph>
      <Paragraph position="5"> Given training data, a machine learning method produces a classifier that outputs possible or impossible according to a given feature vector. We omit the details of ME and SVM. Readers are referred to the above references.</Paragraph>
      <Paragraph position="6"> We will compare three sets of features, F</Paragraph>
      <Paragraph position="8"> . The numbers of the responses tagged possible and impossible were 1,944 and 1,057, respectively. We used 10-fold cross validation to evaluate the accuracies of ME and</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>