<?xml version="1.0" standalone="yes"?>
<Paper uid="N04-1008">
  <Title>Automatic Question Answering: Beyond the Factoid</Title>
  <Section position="7" start_page="11" end_page="11" type="evalu">
    <SectionTitle>
6 Performance issues
</SectionTitle>
    <Paragraph position="0"> In building our system, we have demonstrated that a statistical model can capitalize on large amounts of readily available training data to achieve reasonable performance on answering non-factoid questions. Our system was able to successfully answer a large variety of complex, non-factoid questions, such as: Q: Can I travel with the Ameripass in Mexico? A: The Ameripass includes all of USA excluding Alaska.</Paragraph>
    <Paragraph position="1"> In Canada, it is valid for travel to Montreal, Quebec, Toronto, Ontario, Vancouver BC, and New Westminster BC.</Paragraph>
    <Paragraph position="2"> In Mexico it is valid for travel to Matamoros, Nuevo Laredo, Tamaulipas and Tijuana Baja California.</Paragraph>
    <Paragraph position="3"> Q: What is a PIC Freeze? A: How to Prevent Slamming: Institute a PIC Freeze The best way to keep from being slammed, or having a long distance company change your service without your permission, is to request a &amp;quot;PIC freeze&amp;quot; for your line. PIC, or presubscribed interexchange carrier, is the telecom industry 's term for a long distance company.</Paragraph>
    <Paragraph position="4"> For those questions which were not answered correctly, we identified some of the most frequent causes which led to erroneous answers: * answer was not in the retrieved pages (see the 46% performance ceiling given by the Oracle) * answer was of the wrong &amp;quot;type&amp;quot; (e.g., an answer for &amp;quot;how-to&amp;quot; instead of &amp;quot;what-is&amp;quot;): Q: What are best graduate schools for AI? A: If you are applying for grad school in AI, and you did some research for an AI person, even if it was long ago and you don't consider it to be relevant, get a recommendation if you think that the person likes you at all. [...] * it pointed to where an answer might be instead of answering the question: Q: What do research studies say about massage therapy? A: It supports research on the efficacy of therapeutic massage through the public, charitable AMTA Foundation. Additional information about massage therapy and about AMTA is available via the Web at www.amtamassage.org.</Paragraph>
    <Paragraph position="5"> * the translation model overweighed the answer language model (too good a &amp;quot;translation&amp;quot;, too bad an answer) Q: What are private and public keys? A: Private and public keys Private and public keys Algorithms can use two types of keys: private and public. * did not pick up the key content word (in the example below, eggs) Q: What makes eggs have thin, brittle shells? A: The soft-shelled clams, such as steamer, razor, and geoduck clams, have thin brittle shells that can't completely close. Cod - A popular lean, firm, white meat fish from the Pacific and the North Atlantic.</Paragraph>
    <Paragraph position="6"> It is worth pointing out that most of these errors do not arise from within a single module, but rather they are the result of various interactions between modules that miss on some relevant information.</Paragraph>
  </Section>
class="xml-element"></Paper>