<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-2012">
  <Title>Answer Validation by Keyword Association</Title>
  <Section position="11" start_page="5" end_page="5" type="evalu">
    <SectionTitle>
6.2 Results
</SectionTitle>
    <Paragraph position="0"> Against the development and the test sets, Table 10 shows the results of evaluating the total procedure of keyword selection and answer selection presented in Section 5. The table also shows the performance of baselines:  amount i.e., keyword association ratio presented in Section 3.2.2, and word weights of Section 3.1 + answer selection of Section 4. Integration of keyword association ratio and word weight out-performs those baselines. In total, about 79% (for the development set) and 76% (for the test set) of the questions are solved by the proposed answer validation method based on keyword association. null Comparing the performance of the two data sets, word weights + answer selection has 4% lower precision in the test set. This result indicates that rules for word weights as well as answer selection rules overfit to the development set. On the other hand, the difference of the precisions of the keyword association ratio is much less between the two data sets, indicating that keyword association ratio has less overfit to the development set.</Paragraph>
    <Paragraph position="1"> Finally, the result of the experiment where the development set was solved by the integration method was classified by prize money amount. The result is shown in Figure 1. The more the prize money amount is, the lower the precision seems to be, while their precisions are all above 60%, and their differences are less than 20% in most cases. It can be concluded that our system can solve questions of all the levels almost equally.</Paragraph>
  </Section>
class="xml-element"></Paper>