<?xml version="1.0" standalone="yes"?>
<Paper uid="H05-1117">
  <Title>Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 931-938, Vancouver, October 2005. c(c)2005 Association for Computational Linguistics Automatically Evaluating Answers to Definition Questions</Title>
  <Section position="1" start_page="0" end_page="0" type="abstr">
    <SectionTitle>
Abstract
</SectionTitle>
    <Paragraph position="0"> Following recent developments in the automatic evaluation of machine translation and document summarization, we present asimilarapproach, implementedinameasure called POURPRE, for automatically evaluatinganswerstodefinitionquestions.</Paragraph>
    <Paragraph position="1"> Until now, the only way to assess the correctness of answers to such questions involves manual determination of whether an information nugget appears in a system's response. The lack of automatic methods for scoring system output is an impediment to progress in the field, which we address with this work. Experiments with the TREC 2003 and TREC 2004 QA tracks indicate that rankings produced by our metric correlate highly with official rankings, and that POURPRE outperforms direct application of existing metrics.</Paragraph>
  </Section>
class="xml-element"></Paper>