<?xml version="1.0" standalone="yes"?>
<Paper uid="W96-0104">
  <Title>Learning similarity-based word sense disambiguation from sparse data</Title>
  <Section position="4" start_page="47" end_page="49" type="intro">
    <SectionTitle>
2 Experimental evaluation of the method
</SectionTitle>
    <Paragraph position="0"> We tested the algorithm on the Treebank-2 corpus, which contains 1 million words from the Wall Street Journal, 1989, and is considered a small corpus for the present task. As the MRD, we used a combination of the Webster, the Oxford and the WordNet online dictionaries (the latter used as a thesaurus only). During the development and the tuning of the algorithm, we used the method of pseudo-words (Gale et al., 1992; Schutze, 1992), to save the need for manual verification of the resulting sense tags.</Paragraph>
    <Paragraph position="1"> The final algorithm was tested on a total of 500 examples of four polysemous words: drug, sentence, suit, and player (see Table 1). The relatively small number of polysemous words we studied was dictated by the size and nature of the corpus (we are currently testing additional words, using texts from the British National Corpus).</Paragraph>
    <Paragraph position="2"> The average success rate of our algorithm was 92%. The original training set (before the addition of the feedback sets) consisted of a few dozen examples, in comparison to thousands of examples needed in other corpus-based methods (Schutze, 1992; Yarowsky, 1995).</Paragraph>
    <Paragraph position="3"> Results on two of the words on which we tested our algorithm (drug and suit) have been also reported in the works of Schutze and Yarowsky. It is interesting to compare the performance of the different methods on these words. On the word drug, our algorithm achieved performance of 90.5%, after being trained on 148 examples (contexts). In comparison, (Yarowsky, 1995) achieved  91.4% correct performance, using 1380 contexts and the dictionary definitions in training. 4 On the word suit, our method achieved performance of 94.8%, using 233 training contexts; in comparison, (Schutze, 1992) achieved 957o correct performance, using 8206 contexts. In summary, our algorithm achieved performance comparable to some of the best reported results, using much less data for training. This feature of our approach is important, because the size of the available training set is usually severely constrained for most senses of most words (Gale et al., 1992). Finally, we note that, as in most corpus-based methods, supplying additional examples is expected to improve the performance.</Paragraph>
    <Paragraph position="4"> We now present in detail several of the results obtained with the word drug. A plot of the improvement in the performance vs. iteration number appears in Figure 2. The success rate is plotted for each sense, and for the weighted average of both senses we considered (the weights are proportional to the numb~er of examples of each sense).</Paragraph>
    <Paragraph position="5"> Figure 3 shows how the similarity values develop with iteration number. For each example S of the narcotic sense of drug, the value of sims(S, narcotic) increases with n. Note that after several iterations the similarity values are close to 1, and, because they are bounded by 1, they cannot change significantly with further iterations.</Paragraph>
    <Paragraph position="6"> Figure 4 compares the similarities of a narcotic example to the narcotic sense and to the medicine sense, for each iteration. The medicine sense assignment, made in the first iteration, has been corrected in the following iterations.</Paragraph>
    <Paragraph position="7"> Table 2 shows the most similar words found for the words with the highest weights in the drug example (low-similarity words have been omitted). Note that the similarity is contextual, and is affected by the polysemous target word. For example, traJficking was found to be similar to crime, because in drug contexts the expressions drug trajficking and crime are highly related. In general, traJficking and crime need not be similar, of course.</Paragraph>
    <Paragraph position="8"> 4Yarowsky subsequently improved that result to 93.9%, using his &amp;quot;one sense per discourse&amp;quot; constraint. We expect that a similar improvement could be achieved if that constraint were used in conjunction with our method.  Word Most contextually similar words The medicine sense: medication antibiotic blood prescription medicine percentage pressure prescription analyst antibiotic blood campaign introduction law line-up medication medicine percentage print profit publicity quarter sedative state television tranquilizer use medicine prescription campaign competition dollar earnings law manufacturing margin print product publicity quarter result sale saving sedative staff state television tranquilizer unit use disease antibiotic blood line-up medication medicine prescription symptom hypoglycemia insulin warning manufacturer product plant animal death diabetic evidence finding metabolism study insulin hypoglycemia manufacturer product symptom warning death diabetic finding report study tranquilizer campaign law medicine prescription print publicity sedative television use analyst profit state dose appeal death impact injury liability manufacturer miscarriage refusing ruling diethylstilbestrol hormone damage effect female prospect state The narcotic sense: i consumer distributor effort cessation consumption country reduction requirement victory battle capacity cartel government mafia newspaper people mafia terrorism censorship dictatorship newspaper press brother nothing aspiration assassination editor leader politics rise action country doubt freedom mafioso medium menace solidarity structure trade world terrorism censorship doubt freedom mafia medium menace newspaper press solidarity structure murder capital-punishment symbolism trafficking furor killing substance crime restaurant law bill case problem menace terrorism freedom solidarity structure medium press censorship country doubt mafia newspaper way attack government magnitude people relation threat world trafficking crime capital-punishment furor killing murder restaurant substance symbolism dictatorship aspiration brother editor mafia nothing politics press assassination censorship leader newspaper rise terrorism assassination brother censorship dictatorship mafia nothing press terrorism aspiration editor leader newspaper politics rise laundering army lot money arsenal baron economy explosive government hand materiel military none opinion portion talk censorship mafia newspaper press terrorism country doubt freedom medium menace solidarity structure  are those with the highest weights, whose similarity values have, therefore, the greatest effect. Note that the similarity is contextual, and is highly dependent on the polysemous target word. For example, trafficking was found to be similar to crime, because in the drug contexts the expressions drug trafficking and crime are highly related. In general, trafficking and crime need not be similar, of course. Also note that the similarity is affected by the training corpus. For example, in the Wall Street Journal, the word medicine is mentioned mostly in contexts of making profit, and in advertisements. Thus, in the medicine cluster there one finds words such as analyst, campaign, profit, quarter, dollar, which serve as hints for the medicine sense. Although profit and medicine are not closely related semantically (relative to a more balanced corpus than WSJ), their contexts in the WSJ contain words that are similarly indicative of the sense of the target word. This kind of similarity, therefore, suits its purpose, which is sense disambiguation, although it may run counter to some of our intuitions regarding general semantic similarity.</Paragraph>
  </Section>
class="xml-element"></Paper>