<?xml version="1.0" standalone="yes"?>
<Paper uid="A00-1013">
  <Title>DP: A Detector for Presuppositions in survey questions Katja WIEMER-HASTINGS Psychology Department / Institute for Intelligent Systems</Title>
  <Section position="1" start_page="0" end_page="92" type="abstr">
    <SectionTitle>
Abstract
</SectionTitle>
    <Paragraph position="0"> This paper describes and evaluates a detector of presuppositions (DP) for survey questions.</Paragraph>
    <Paragraph position="1"> Incorrect presuppositions can make it difficult to answer a question correctly.</Paragraph>
    <Paragraph position="2"> Since they can be difficult to detect, DP is a useful tool for questionnaire designer. DP performs well using local characteristics of presuppositions. It reports the presupposition to the survey methodologist who can determine whether the presupposition is valid.</Paragraph>
    <Paragraph position="3"> Introduction Presuppositions are propositions that take some information as given, or as &amp;quot;the logical assumptions underlying utterances&amp;quot; (Dijkstra &amp; de Smedt, 1996, p. 255; for a general overview, see McCawley, 1981). Presupposed information includes state of affairs, such as being married; events., such as a graduation; possessions, such as a house, children, knowledge about something; and others. For example, the question, &amp;quot;when did you graduate from college&amp;quot;, presupposes the event that the respondent did in fact graduate from college. The answer options may be ranges of years, such as &amp;quot;between 1970 and 1980&amp;quot;. Someone who has never attended college can either not respond at all, or give a random (and false) reply. Thus, incorrect presuppositions cause two problems. First, the question is difficult to answer. Second, assuming that people feel obliged to answer them anyway, their answers present false information. This biases survey statistics, or, in an extreme case, makes them useless.</Paragraph>
    <Paragraph position="4"> The detector for presuppositions (DP) is part of the computer tool QUAID (Graesser, Wiemer-Hastings, Kreuz, Wiemer-Hastings &amp; Marquis, in press), which helps survey methodologists design questions that are easy to process. DP detects a presupposition and reports it to the survey methodologist, who can examine if the presupposition is correct. QUAID is a computerized QUEST questionnaire evaluation aid. It is based on QUEST (Graesser &amp; Franklin, 1990), a computational model of the cognitive processes underlying human question answering.</Paragraph>
    <Paragraph position="5"> QUAID critiques questions with respect to unfamiliar technical terms, vague terms, working memory overload, complex syntax, incorrect presuppositions, and unclear question purpose or category. These problems are a subset of potential problems that have been identified by Graesser, Bommareddy, Swamer, and Golding (1996; see also Graesser, Kennedy, Wiemer-Hastings &amp; Ottati, 1999).</Paragraph>
    <Paragraph position="6"> QUAID performs reliably on the first five problem categories. In comparison to these five problems, presupposition detection is even more challenging. For unfamiliar technical terms, for example, QUAID reports words with frequencies below a certain threshold. Such an elegant solution is impossible for presuppositions. Their forms vary widely across presupposition types. Therefore, their detection requires a complex set of rules, carefully tuned to identify a variety of presupposition problems. DP prints out the  presuppositions of a question, and relies on the survey methodologist to make the final decision whether the presuppositions are valid.</Paragraph>
    <Paragraph position="7"> 1 How to detect presuppositions We conducted a content analysis of questions with presupposition problems to construct a list of indicators for presuppositions. 22 questions containing problematic presuppositions were selected from a corpus of 550 questions, taken from questionnaires provided by the U.S. Census Bureau. The 22 questions were identified based on ratings by three human expert raters. It may seem that this problem is infrequent, but then, these questions are part of commonly used questionnaires that have been designed and revised very thoughtfully.</Paragraph>
    <Paragraph position="8"> Additionally, we randomly selected a contrast question sample of 22 questions rated unproblematic with regard to incorrect presuppositions by all three raters. Examples (1) and (2) are questions rated as problematic by at least two raters; examples (3) and (4) present questions that do not contain presuppositions.</Paragraph>
    <Paragraph position="9">  (1) Is that the same place you USUALLY go when you need routine or preventive care, such as a physical examination or check up? (2) How much do your parents or parent know about your close friends' parents? (3) From date to December 31, did you take one or more trips or outings in the United States, of at least one mile, for the PRIMARY purpose of observing, photographing, or feeding wildlife? (4) Are you now on full-time active duty with the armed forces?  Example (1) presupposes the habit of making use of routine / preventive care; (2) presupposes that the respondent has close friends.</Paragraph>
    <Paragraph position="10"> As stated above, incorrect presuppositions are infrequent in well-designed questionnaires. For example, questions about details of somebody's marriage are usually preceded by a question establishing the person's marital status.</Paragraph>
    <Paragraph position="11"> In spite of this, providing feedback about presuppositions to the survey methodologist is useful. Importantly, QUAID is designed to aid in the design process. Consider a survey on healthrelated issues. In the context of this topic, a survey methodologist may be interested in how many days of work a person missed because of illness, but not think about whether the person actually has a job. Upon entering the question &amp;quot;how many days of work did you miss last year because of illness&amp;quot; into the QUAID tool, DP would report that the question presupposes employment. The survey methodologist could then insert a question about employment.</Paragraph>
    <Paragraph position="12"> Second, there are subtle presuppositions that may go undetected even by a skilled survey designer. These are presuppositions about things that are likely (but not necessarily) true. For example, a question may inquire about a person's close friends (presupposing close friends) or someone's standard place for preventive care (presupposing the habit of making use of preventive care). DP does not know which presuppositions are likely to be valid or invalid, and is therefore more likely to detect such subtle incorrect presuppositions than a human expert.</Paragraph>
    <Section position="1" start_page="90" end_page="92" type="sub_section">
      <SectionTitle>
1.1 The presupposition detector (DP)
</SectionTitle>
      <Paragraph position="0"> We constructed a set of presupposition detection rules based on the content analysis. The rules use a wide range of linguistic information about the input sentences, including particular words (such as &amp;quot;why&amp;quot;), part of speech categories (e.g., whpronoun), and complex syntactic subtrees (such as a quantification clause, followed by a noun phrase).</Paragraph>
      <Paragraph position="1"> 1.1.1 The syntactic analysis component We used Eric Brill's rule-based word tagger (1992, 1994a, 1994b), the de facto state of the art tagging system, to break the questions down into part-of-speech categories. Brill's tagger produces a single lexical category for each word in a sentence by first assigning tags based on the frequency of occurrence of the word in that category, and then applying a set of context-based re-tagging rules.</Paragraph>
      <Paragraph position="2"> The tagged text was then passed on to Abney's SCOL/CASS system (1996a, 1996b), an extreme bottom-up parser. It is designed to avoid ambiguity problems by applying grammar rules on a level-by-level basis. Each level contains rules that will only fire if they are correct with high probability. Once the parse moves on to a higher level, it will not attempt to apply lower-level rules. In this way, the parser identifies chunks of information, which it can be reasonably certain are  connected, even when it cannot create a complete parse of a sentence.</Paragraph>
      <Paragraph position="3"> 1.1.2 The presupposition indicators The indicators for presuppositions were tested against questions rated as &amp;quot;unproblematic&amp;quot; to eliminate items that failed to discriminate questions with versus without presuppositions.</Paragraph>
      <Paragraph position="4"> We constructed a second list of indicators that detect questions containing no presuppositions.</Paragraph>
      <Paragraph position="5"> All indicators are listed in Table 1. These lists are certainly far from complete, but they present a good basis for evaluating of how well presuppositions can be detected by an NLP system. These rules were integrated into a decision tree structure, as illustrated in Figure 1.</Paragraph>
    </Section>
    <Section position="2" start_page="92" end_page="92" type="sub_section">
      <SectionTitle>
1.2 Classifying presuppositions
</SectionTitle>
      <Paragraph position="0"> Different types of presuppositions can be distinguished based on particular indicators.</Paragraph>
      <Paragraph position="1"> Examples for presupposition types, such as events or possessions, were mentioned above.</Paragraph>
      <Paragraph position="2"> Table 2 presents an exhaustive overview of presupposition types identified in our analysis. Note that some indicators can point to more than one type of presupposition.</Paragraph>
      <Paragraph position="3"> Table 2 : Classification of presupposition based on indicators. In the right column, expressions in parentheses identify the presupposed unit.</Paragraph>
      <Paragraph position="4">  an a~ent (A person who VP) an event (VP) DP reports when a presupposition is present, and it also indicates the type of presupposition that is made (e.g., a common ground presupposition or the presupposition of a habit) in order to point the question designer to the potential presupposition error. DP uses the expressions in the right column in Table 2, selected in accordance with the indicators, and fills them into the brackets in its output (see Figure 1). For example, given the question &amp;quot;How old is your child?&amp;quot;, DP would detect the possessive pronoun &amp;quot;your&amp;quot;, and accordingly respond: &amp;quot;It looks like you are presupposing a possession (child). Make sure that the presupposition is correct by consulting the previous questions.&amp;quot;</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>