<?xml version="1.0" standalone="yes"?>
<Paper uid="H05-1087">
  <Title>Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 692-699, Vancouver, October 2005. c(c)2005 Association for Computational Linguistics Maximum Expected F-Measure Training of Logistic Regression Models</Title>
  <Section position="7" start_page="697" end_page="697" type="concl">
    <SectionTitle>
8 Conclusions
</SectionTitle>
    <Paragraph position="0"> We have presented a novel estimation procedure for probabilistic classifiers which we call, by a slight abuse of terminology, maximum expected F-measure training. We made use of the fact that expected utility computations can be carried out in a vector space, and that an ordering of vectors can be imposed for purposes of maximization which can employ auxiliary functions like the F-measure (2).</Paragraph>
    <Paragraph position="1"> This technique is quite general and well suited for working with other quantities that can be expressed in terms of hits, misses, false alarms, correct rejections, etc. In particular, it could be used to find a point estimate which provides a certain tradeoff between specificity and sensitivity, or operating point.</Paragraph>
    <Paragraph position="2"> A more general method would try to optimize several such operating points simultaneously, an issue which we will leave for future research.</Paragraph>
    <Paragraph position="3"> The classifiers discussed in this paper are logistic regression models. However, this choice is not crucial. The approximation (4) is reasonable for binary decisions in general, and one can use it in conjunction with any well-behaved conditional Bernoulli model or related classifier. For Support Vector Machines, approximate F-measure maximization was introduced by Musicant et al. (2003).</Paragraph>
    <Paragraph position="4"> Maximizing F-measure during training seems especially well suited for dealing with skewed classes.</Paragraph>
    <Paragraph position="5"> This can happen by accident, because of the nature of the problem as in our summarization example above, or by design: for example, one can expect skewed binary classes as the result of the one-vs-all reduction of multi-class classification to binary classification; and in multi-stage classification one may want to alternate between classifiers with high recall and classifiers with high precision.</Paragraph>
    <Paragraph position="6"> Finally, the ability to incorporate non-standard tradeoffs between precision and recall at training time is useful in many information extraction and retrieval applications. Human end-users often create asymmetries between precision and recall, for good reasons: they may prefer to err on the side of caution (e.g., it is less of a problem to let an unwanted spam email reach a user than it is to hold back a legitimate message), or they may be better at some tasks than others (e.g., search engine users are good at filtering out irrelevant documents returned by a query, but are not equipped to crawl the web in order to look for relevant information that was not retrieved). In the absence of methods that work well for a wide range of operating points, we need training procedures that can be made sensitive to rare cases depending on the particular demands of the application.</Paragraph>
  </Section>
class="xml-element"></Paper>