<?xml version="1.0" standalone="yes"?>
<Paper uid="W06-0705">
  <Title>Using Scenario Knowledge in Automatic Question Answering</Title>
  <Section position="5" start_page="32" end_page="34" type="metho">
    <SectionTitle>
CASE 1:
Scenario:
</SectionTitle>
    <Paragraph position="0"> companies could gain from relocating to India the types of economic advantages that American Answer: Question: GE and Dell have reported earnings growth after outsourcing jobs to both Indonesia and India What U.S. companies are outsourcing jobs to Indonesia?</Paragraph>
    <Section position="1" start_page="32" end_page="34" type="sub_section">
      <SectionTitle>
Textual Entailment Contextual Entailment
Scenario:
</SectionTitle>
      <Paragraph position="0"> companies could gain from relocating to India the types of economic advantages that American Answer: Question: Scenario: companies could gain from relocating to India the types of economic advantages that American Answer: Question: certain types of jobs to India? How could U.S. companies profit from moving How could U.S. companies benefit by moving jobs to India? Outsourcing jobs to India saved the carrier $25 million, enabling it to turn a profit for the first time. Despite public opposition to outsourcing jobs to India, political support has never been higher.  In Case 1, the scenario textually entails the meaning of the answer passage, as earnings growth from outsourcing necessarily represents one of the types of economic advantages that can be derived from outsourcing. However, the scenario cannot be seen as entailing the user's question, as the user's interest in job outsourcing in Indonesia cannot be interpreted as being part of the topics that are associated with the scenario. In this case, recognition of contextual entailment would allow systems to be sensitive to the types of  scenario-relevant information that is encountered even when the user asks questions that are not entailed by the scenario itself. We expect that this type of contextual entailment would allow systems to identify scenario-relevant knowledge throughout a user's interaction with a system, regardless of topic of a user's last query.</Paragraph>
      <Paragraph position="1"> In Case 2, the user's question is entailed by the scenario, but no corresponding entailment relationship can be established between the scenario and the answer passage identi ed by the Q/A system as an answer to the question. While political support may be interpretable as one of the bene ts realized by companies that outsource, it cannot be understood as one of the economic advantages of outsourcing. Here, recognizing that contextual entailment could not be established between the scenario and the answer but could be established between the scenario and the question could be used to signal the Q/A system to consider additional answers before moving on to the user's next question. By identifying contextual entailment relationships between answers and elements in a scenario, systems could perform valuable forms of answer validation that could be used to select only the most relevant answers for a user's consideration. null Finally, in Case 3, entailment relationships exist between the scenario and both the user's question and the returned answer, as saving $25 million can be considered to be both an economic advantage and one of the ways that companies pro t from outsourcing. In this case, the establishment of contextual entailment could be used to inform topic models that could be used to identify and extract other similarly relevant passages for the user. In order to capture these three types of CE relationships, we developed the architecture for recognizing contextual entailment illustrated in Figure 3.</Paragraph>
      <Paragraph position="2"> This architecture includes three basic types of modules: (1) a Context Discovery module, which identi es passages relevant to the concepts mentioned in a scenario, (2) a Textual Entailment module, which recognizes implicational relationships between passages, and (3) a Entailment Merging module, which ranks relevant passages according to their relevance to the scenario itself. In Context Discovery, document retrieval queries are rst extracted from each sentence found in a scenario. Once a set of documents has been as- null sembled, topic signatures (Lin and Hovy, 2000; Harabagiu 2004) are computed which identify the set of topic-relevant concepts and relations between concepts that are found in the relevant set of documents. Weights associated with each set of topic signatures are then used to extract a set of relevant sentences referred to as topic answers from each relevant document. Once a set of topic answers have been identi ed, each topic answer is paired with a question submitted by a user and sent to the Textual Entailment system described in Section 2. Topic answers that are deemed to be positive entailments of the user question are assigned a con dence value by the TE system and are then sent to a Entailment Merging module, which uses logistic regression in order to rank passages according to their expected relevance to the user scenario. Here, logistic regression is used to nd a set of coef cients bj (where 0 [?] j [?] p) in order to t a variable x to a logistic transformation of a probability q.</Paragraph>
      <Paragraph position="4"> We believe that since logistic regression uses a maximum likelihood method, it is a suitable technique for normalizing across range of con dence values output by the TE system.</Paragraph>
    </Section>
  </Section>
  <Section position="6" start_page="34" end_page="34" type="metho">
    <SectionTitle>
3 Recognizing Textual Entailment
</SectionTitle>
    <Paragraph position="0"> Recent work in computational semantics (Haghighi et al., 2005; Hickl et al., 2006b; MacCartney et al., 2006) has demonstrated the viability of supervised machine learning-based approaches to the recognition of textual entailment (TE). While these approaches have not incorporated the forms of structured world knowledge featured in many logic-based TE systems, classi cation-based approaches have been consistently among the top-performing systems in both the 2005 and 2006 PASCAL Recognizing Textual Entailment (RTE) Challenges (Dagan et al., 2005), with the best systems (such as (Hickl et al., 2006b)) correctly identifying instances of textual entailment more than 75% of the time.</Paragraph>
    <Paragraph position="1"> The architecture of our TE system is presented in Figure 4.1 Pairs of texts are initially sent to a Preprocessing Module, which performs syntactic and semantic parsing of each sentence, resolves coreference, and annotates entities and predicates with a wide range of lexico-semantic and prag1For more information on the TE system described in this section, please see (Hickl et al., 2006b) and (Harabagiu and Hickl, 2006).</Paragraph>
    <Paragraph position="2"> matic information, including named entity information, synonymy and antonymy information, and polarity and modality information.</Paragraph>
    <Paragraph position="3"> Once preprocessing is complete, texts are then sent to an Alignment Module, which uses lexical alignment module in conjunction with a paraphrase acquisition module in order to determine the likelihood that pairs of elements selected from each sentence contain corresponding information that could be used in recognizing textual entailment. Lexical Alignment is performed using a Maximum Entropy-based classi er which computes an alignment probability p(a) equal to the likelihood that a term selected from one text corresponds to an element selected from another text. Once these pairs of corresponding elements are identi ed, alignment information is then used in order to extract portions of texts that could be related via one or more phrase-level alternations or paraphrases . In order to acquire these alternations, the most likely pairs of aligned elements were then sent to a Paraphrase Acquisition module, which extracts sentences that contain instances of both aligned elements from the World Wide Web.</Paragraph>
    <Paragraph position="4"> Output from these two modules are then combined in a nal Classi cation Module, which uses features derived from (1) lexico-semantic properties, (2) semantic dependencies, (3) predicate-based features (including polarity and modality), (4) lexical alignment, and (5) paraphrase acquisition in order learn a decision tree classi er capable of determining whether an entailment relationship exists for a pair of texts.</Paragraph>
  </Section>
  <Section position="7" start_page="34" end_page="35" type="metho">
    <SectionTitle>
4 Intrinsic Evaluation of Contextual
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="34" end_page="35" type="sub_section">
      <SectionTitle>
Entailment
</SectionTitle>
      <Paragraph position="0"> Since we believe CE is intrinsic to the Q/A task, we have evaluated the impact of contextual entailment on our Q/A system in two ways. First, we compared the quality of the answers produced, with and without contextual entailment. Second, we evaluated the quality of the ranked list of paragraphs against the list of entailed paragraphs identi ed by the CE system and the set of relevant answers identi ed by the Q/A system. This comparison was performed for each of the three cases of entailment presented in Figure 2.</Paragraph>
      <Paragraph position="1"> We have explored the impact of knowledge derived from the user scenario through different forms of contextual entailment by comparing the  results of such knowledge integration in a Q/A system against the usage of scenario knowledge reported in (Harabagiu et al., 2005).</Paragraph>
      <Paragraph position="2"> Topic signatures, derived from the user scenario and from documents are used to establish text passages that are relevant to the scenario, and thus constitute relevant answers. For each such answer, one or multiple questions were built automatically with the method reported in (Harabagiu et al., 2005). When a new question was asked, its similarity to any of the questions generated based on the knowledge of the scenario is computed, and its corresponding answer is provided as an answer for the current question as well. Since the questions are ranked by similarity to the current question, the answers are also ranked and produce the Answer Set1 illustrated in Figure 5.</Paragraph>
      <Paragraph position="3"> When a Q/A system is used for answering the question, the scenario knowledge can be used in two ways. First, the keywords extracted by the Question Processing module can be enhanced with concepts from the topic signatures to produce a ranked list of paragraphs, resulting from the Passage Retrieval Module. These passages together with the question and the user scenario are used in one of the contextual entailment con gurations to derive a list of entailed paragraphs from which the Answer Processing module can extract the answer set 2 illustrated in Figure 5. In another way, the ranked list of paragraphs is passed to the Answer Processing module, which provides a set of ranked answers to the contextual entailment congurations to derive a list of entailed answers, represented as answer set 3 in Figure 5. We evaluate the quality of each set of answers, and for the answer set 2 and 3, we produce separate evaluation for each con guration for the contextual entailment. null</Paragraph>
    </Section>
  </Section>
  <Section position="8" start_page="35" end_page="36" type="metho">
    <SectionTitle>
5 Extrinsic Evaluation of Contextual
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="35" end_page="36" type="sub_section">
      <SectionTitle>
Entailment
</SectionTitle>
      <Paragraph position="0"> Questions asked in response to a user scenario tend to be complex. Following work in (Hickl et al., 2004), we believe complex questions can be answered in one of two ways: either by (1) using techniques (similar to the ones proposed in (Harabagiu et al., 2006)) for automatically decomposing complex questions into sets of informationally-simpler questions, or by (2) using a multi-document summarization (MDS) system (such as the one described in (Lacatusu et al., 2006)) in order to assemble a ranked list of passages which contain information that is potentially relevant to the user's question.</Paragraph>
      <Paragraph position="1"> First, we expect that contextual entailment can be used to select the decompositions of a complex question that are most closely related to a scenario. By assigning more con dence to the decompositions that are contextually entailed by a scenario, systems can select a set of answers that are relevant to both the user scenario and the user's question. In contrast, contextual entailment can be used in conjunction with the output of a MDS system: once a summary has been constructed from the passages retrieved for a query, contextual en- null The architecture of this proposed system is illustrated in Figure 6.</Paragraph>
      <Paragraph position="2"> When using contextual entailment for selecting question decompositions, we rely on the method reported in (Harabagiu et al., 2006) which generates questions by using a random walk on a bipartite graph of salient relations and answers. In this case, the recognition of entailment between questions operates as a lter, forcing questions that are not entailed by any of the signature answers derived from the scenario context (see Figure 3) to be dropped from consideration.</Paragraph>
      <Paragraph position="3"> When entailment information is used for re-ranking candidate answers, the summary is added to the scenario context, each summary sentence being treated akin to a signature answer. We believe that the summary contains the most informative information from both the question and the scenario, since the queries that produced it originated both in the question and in the scenario. By adding summary sentences to the scenario context, we have introduced a new dimension to the processing of the scenario. The contextual entailment is based on the textual entailments between any of the texts from the scenario context and any of the candidate answers.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>