<?xml version="1.0" standalone="yes"?>
<Paper uid="C04-1136">
  <Title>Significance tests for the evaluation of ranking methods</Title>
  <Section position="4" start_page="0" end_page="0" type="concl">
    <SectionTitle>
5 Conclusion
</SectionTitle>
    <Paragraph position="0"> In the past, various statistical tests have been used to assess the significance of results obtained in the evaluation of ranking methods. There is much confusion about their validity, though, mainly due to 15The agreement is confirmed by the Kolmogorov test of goodness-of-fit, which does not reject the theoretical model (4) in either case.</Paragraph>
    <Paragraph position="1">  G2 (left panel) and t (right panel). The solid lines indicate the expected distribution according to Eq. (2). the fact that assumptions behind the application of a test are seldom made explicit. This paper is an attempt to remedy the situation by interpreting the evaluation procedure as a random experiment. The model assumptions, motivated by intuitive arguments, are stated explicitly and are open for discussion. Empirical validation on a collocation extraction task has confirmed the usefulness of the model, indicating that it represents a lower bound on the variability of evaluation results. On the basis of this model, I have developed appropriate significance tests for the evaluation of ranking methods. These tests are implemented in the UCS toolkit, which was used to produce the graphs in this paper and can be downloaded from http: //www.collocations.de/.</Paragraph>
  </Section>
class="xml-element"></Paper>