<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-2904">
  <Title>Scoring Algorithms for Wordspotting Systems</Title>
  <Section position="4" start_page="0" end_page="0" type="metho">
    <SectionTitle>
3 Performance Measures
</SectionTitle>
    <Paragraph position="0"> We propose two scoring evaluation measures. In each of these methods, the raw score is modified by some scoring function F(). The first measure evaluates a scoring algorithms usefulness for setting detection thresholds. This method assumes that the scoring function calculates the cdf of the missed score distributions. The measurement is based on the Kolmogorov-Smirnov test statistic, which is given by</Paragraph>
    <Paragraph position="2"> where R(i)M are the raw scores for the false alarms in descending order.</Paragraph>
    <Paragraph position="3"> A metric for measuring scoring algorithms based on result confidence is given by</Paragraph>
    <Paragraph position="5"> where NM and NH are the number of hits and misses.</Paragraph>
    <Paragraph position="6"> This value is equal to zero when all hits are scored to one and all misses are scored as zero. On the other hand, B is equal to 0.5 if F(R) is set to 0.5 regardless of the input.</Paragraph>
  </Section>
  <Section position="5" start_page="0" end_page="0" type="metho">
    <SectionTitle>
4 Algorithms
</SectionTitle>
    <Paragraph position="0"> If one is interested in setting a detection threshold based on false alarms per hour, then one can set the score using the cumulative density function of the misses. This yields</Paragraph>
    <Paragraph position="2"> whereQisthecdfoftheunitnormaldistribution. Toseta threshold for K false alarms per hour, then the threshold should be set to</Paragraph>
    <Paragraph position="4"> where KT is the range of false alarms per hour that the miss model is trained.</Paragraph>
    <Paragraph position="5"> If one is looking at a list of scores, one might be interested in the probability that the score was generated by a true match. By Bayes law, the conditional probability can be calculated by</Paragraph>
    <Paragraph position="7"> where PH is the prior probability of a hit.</Paragraph>
  </Section>
  <Section position="6" start_page="0" end_page="0" type="metho">
    <SectionTitle>
5 Model Training
</SectionTitle>
    <Paragraph position="0"> Each of the scoring methods described above require models of how the phonemes relate to the scores through the parameters: uM, uH, s2M, and s2H. For this purpose, a series of hits and misses over the desired range of false alarmsratesmustbecollectedfromthewordspotter. With these scores, it is possible to train the miss and hit models independently. For this reason, only the miss model training is described here.</Paragraph>
    <Paragraph position="1"> Given the model in Equation 5, the following distribution holds with N observations:  The maximum likelihood solution for uM and s2M is a difficult optimization problem. However, if the phoneme components R(n)l from Equation 1, the distribution simplifies to observations of the Gaussian components. By using the Expectation Maximization (EM) algorithm, the overall likelihood in Equation 11 can be iteratively maximized (Dempster et al., 1977).</Paragraph>
    <Paragraph position="2"> Similarly, the training problem can also be viewed in a Bayesian framework, where a Minimum Mean Squared Error (MMSE) estimate can be calculated. Like the maximum likelihood estimate, this requires an iterative method where the components of the score are generated. This can be computed by a Gibbs sampler (Gamerman, 1997).</Paragraph>
    <Paragraph position="3"> In addition to providing a mechanism for creating meaningful scores, these models can be useful for other purposes. For example, one can analyze the mean vectors to determine which phonemes provide better discrimination for wordspotting. These can also be used to diagnose problems in performance that are phoneme specific.</Paragraph>
  </Section>
class="xml-element"></Paper>