<?xml version="1.0" standalone="yes"?>
<Paper uid="W00-0712">
  <Title>Knowledge-Free Induction of Morphology Using Latent Semantic Analysis</Title>
  <Section position="5" start_page="70" end_page="71" type="evalu">
    <SectionTitle>
4 Results
</SectionTitle>
    <Paragraph position="0"> We compare our algorithm to Goldsmith's Linguistica (2000) by using CELEX's (Baayen, et al., 1993) suffixes as a gold standard.</Paragraph>
    <Paragraph position="1"> CELEX is a hand-tagged, morphologicallyanalyzed database of English words. CELEX has limited coverage of the words from our data set (where our data consists of over eight million words from random subcollections of TREC data (Voorhees, et a1,1997/8)), so we only considered words with frequencies of 10 or more.</Paragraph>
    <Paragraph position="2">  Morphological relationships can be represented graphically as directed graphs (see Figure 3, where three separate graphs are depicted). Developing a scoring algorithm to compare directed graphs is likely to be prone to disagreements. Therefore, we score only the vertex sets of directed graphs. We will refer to these vertex sets as conflation sets. For example, concern's conflation set contains itself as well as &amp;quot;concerned,&amp;quot; &amp;quot;concerns,&amp;quot; and &amp;quot;concerning&amp;quot; (or, in shorthand notation, the set is {a,b,c,d}).</Paragraph>
    <Paragraph position="3"> To evaluate an algorithm, we sum the number of correct (C), inserted (Z), and deleted (D) words it predicts for each hypothesized conflation set. If Xw represents word w's conflation set according to the algorithm, and if Yw represents its CELEX-based conflation set, then</Paragraph>
    <Paragraph position="5"> However, in making these computations, we disregard any CELEX words that are not in the algorithm's data set and vice versa.</Paragraph>
    <Paragraph position="6"> For example, suppose two algorithms were being compared on a data set where all the words from Figure 3 were available except &amp;quot;concerting&amp;quot; and &amp;quot;concertos.&amp;quot; Suppose further that one algorithm proposed that {a,b,c,d,e,f,g,i} formed a single conflation set whereas the other algorithm proposed the three sets {a,b,c,d},{e,g,i}, and {f}. Then Table 3 illustrates how the two algorithms would be scored.</Paragraph>
    <Paragraph position="7">  To explain Table 3, consider algorithm one's entries for 'a.' Algorithm one had proposed that Xa={a,b,c,d,e,f,g,i} when in reality, Ya={a,b,c,d}. Since IXa NYal = 4 and IYal=4, then CA=4/4. The remaining values of the table can be computed accordingly.</Paragraph>
    <Paragraph position="8"> Using the values from Table 3, we can also compute precision, recall, and F-Score.</Paragraph>
    <Paragraph position="9"> Precision is defined to be C/(C+Z), recall is C/(C+D), and F-Score is the product of precision and recall divided by the average of the two. For the first algorithm, the precision, recall, and F-Score would have respectively been 1/3, 1, and 1/2. In the second algorithm, these numbers would have been 5/7, 5/6, and 10/13.</Paragraph>
    <Paragraph position="10"> Table 4 uses the above scoring mechanism to compare between Linguistica and our system (at various probability thresholds). Note that since Linguistica removes capitalization, it will have a different total word count than our system.</Paragraph>
  </Section>
class="xml-element"></Paper>