<?xml version="1.0" standalone="yes"?>
<Paper uid="C04-1162">
  <Title>PageRank on Semantic Networks, with Application to Word Sense Disambiguation</Title>
  <Section position="8" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
6 Experimental Evaluation
</SectionTitle>
    <Paragraph position="0"> We evaluate the accuracy of the word sense disambiguation algorithms on a benchmark of sense-annotated texts, in which each open-class word is mapped to the meaning selected by a lexicographer as being the most appropriate one in the context of a sentence. We are using a subset of the SemCor texts (Miller et al., 1993) - five randomly selected files covering different topics: news, sports, entertainment, law, and debates - as well as the data set provided for the English all words task during SENSEVAL-2.</Paragraph>
    <Paragraph position="1"> The average size of a file is 600-800 open class words. On each file, we run two sets of evaluations.</Paragraph>
    <Paragraph position="2"> (1) One set consisting of the basic &amp;quot;uninformed&amp;quot; version of the knowledge-based algorithms, where the sense ordering provided by the dictionary is not taken into account at any point. (2) A second set of experiments consisting of &amp;quot;informed&amp;quot; disambiguation algorithms, which incorporate the sense order rendered by the dictionary.</Paragraph>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
6.1 Uninformed Algorithms
</SectionTitle>
      <Paragraph position="0"> Given that word senses are ordered in WordNet by decreasing frequency of their occurrence in large sense annotated data, we explicitly remove this ordering by applying a random permutation of the senses with uniform distribution. This randomization step ensures that any eventual bias introduced by the sense ordering is removed, and it enables us to evaluate the impact of the disambiguation algorithm when no information about sense frequency is available. In this setting, the following dictionary-based algorithms are evaluated and compared: PageRank, Lesk, combined PageRank-Lesk, and the random baseline: PageRank. The algorithm introduced in this paper, which selects the most likely sense of a word based on the PageRank score assigned to the synsets corresponding to the given word within the text graph. While experiments were performed using all semantic relations listed in Sections 4.2 and 4.3, we report here on the results obtained with the xlink relation, which was found to perform best as compared to other semantic relations.</Paragraph>
      <Paragraph position="1"> Lesk. We are also experimenting with the Lesk algorithm described in section 5.1, which decides on the correct sense of a word based on the highest  Sense (WordNet sense order integrated) overlap between the dictionary sense definitions and the context where the word occurs.</Paragraph>
      <Paragraph position="2"> PageRank + Lesk. The PageRank and Lesk algorithms can be combined into one hybrid algorithm, as described in section 5.3. First, we order the senses based on the score assigned by the the Lesk algorithm, and then apply PageRank on this reordered set of senses.</Paragraph>
      <Paragraph position="3"> Random. Finally, we are running a very simple sense annotation algorithm, which assigns a random sense to each word in the text, and which represents a baseline for this set of &amp;quot;uninformed&amp;quot; word sense disambiguation algorithms.</Paragraph>
      <Paragraph position="4"> Table 1 lists the disambiguation precision obtained by each of these algorithms on the evaluation benchmark. On average, PageRank gives an accuracy of 47.27%, which brings a significant 7.7% error reduction with respect to the Lesk algorithm, and 19.0% error reduction over the random baseline.</Paragraph>
      <Paragraph position="5"> The best performance is achieved by a combined PageRank and Lesk algorithm: 51.16% accuracy, which brings a 28.5% error reduction with respect to the random baseline. Notice that all these algorithms rely exclusively on information drawn from dictionaries, and do not require any information on sense frequency, which makes them highly portable to other languages.</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
6.2 Informed Algorithms
</SectionTitle>
      <Paragraph position="0"> In a second set of experiments, we allow the disambiguation algorithms to incorporate the sense order provided by WordNet. While this class of algorithms is informed by the use of global frequency information, it does not use any specific corpus annotations and therefore it leans in gray area between supervised and unsupervised methods.</Paragraph>
      <Paragraph position="1"> We are again evaluating four different algorithms: PageRank, Lesk, combined PageRank - Lesk, and a baseline consisting of assigning by default the most frequent sense.</Paragraph>
      <Paragraph position="2"> PageRank. The PageRank-based algorithm introduced in this paper, combined with the WordNet sense frequency, as described in Section 5.4.</Paragraph>
      <Paragraph position="3"> Lesk. The Lesk algorithm described in section 5.1, applied on an ordered set of senses. This means that words that have two or more senses with a similar score identified by Lesk, will keep the WordNet sense ordering.</Paragraph>
      <Paragraph position="4"> PageRank + Lesk. A hybrid algorithm, that combines PageRank, Lesk, and the dictionary sense order. This algorithm consists of the method described in Section 5.3, applied on the ordered set of senses. Most frequent sense. Finally, we are running a simple &amp;quot;informed&amp;quot; sense annotation algorithm, which assigns by default the most frequent sense to each word in the text (i.e. sense number one in WordNet).</Paragraph>
      <Paragraph position="5"> Table 2 lists the accuracy obtained by each of these informed algorithms on the same benchmark.</Paragraph>
      <Paragraph position="6"> Again, the PageRank algorithm exceeds the other knowledge-based algorithms by a significant margin: it brings an error rate reduction of 21.3% with respect to the most frequent sense baseline, and a 7.2% error reduction over the Lesk algorithm. Interestingly, combining PageRank and Lesk under this informed setting does not bring any significant improvements over the individual algorithms: 67.72% obtained by the combined algorithm compared with 67.51% obtained with PageRank only.</Paragraph>
    </Section>
    <Section position="3" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
6.3 Discussion
</SectionTitle>
      <Paragraph position="0"> Regardless of the setting - fully unsupervised algorithms with no a-priori knowledge about sense order, or informed methods where the sense order rendered by the dictionary is taken into account - the PageRank-based word sense disambiguation algorithm exceeds the baseline by a large margin, and always outperforms the Lesk algorithm. Moreover, a hybrid algorithm that combines the PageRank and Lesk methods into one single algorithm is found to improve over the individual algorithms in the first setting, but brings no significant changes when the sense frequency is also integrated into the disambiguation algorithm. This may be explained by the fact that the additional knowledge element introduced by the sense order in WordNet increases the redundancy of information in these two algorithms to the point where their combination cannot improve over the individual algorithms.</Paragraph>
      <Paragraph position="1"> The most closely related method is perhaps the lexical chains algorithm (Morris and Hirst, 1991) where threads of meaning are identified throughout a text. Lexical chains however only take into account possible relations between concepts in a static way, without considering the importance of the concepts that participate in a relation, which is recursively determined by PageRank. Another related line of work is the word sense disambiguation algorithm proposed in (Veronis and Ide, 1990), where a large neural network is built by relating words through their dictionary definitions.</Paragraph>
      <Paragraph position="2"> The Analogy. In the context of Web surfing, PageRank implements the &amp;quot;random surfer model&amp;quot;, where a user surfs the Web by following links from any given Web page. In the context of text meaning, PageRank implements the concept of text cohesion (Halliday and Hasan, 1976), where from a certain concept C in a text, we are likely to &amp;quot;follow&amp;quot; links to related concepts - that is, concepts that have a semantic relation with the current concept C.</Paragraph>
      <Paragraph position="3"> Intuitively, PageRank-style algorithms work well for finding the meaning of all words in open text because they combine together information drawn from the entire text (graph), and try to identify those synsets (vertices) that are of highest importance for the text unity and understanding.</Paragraph>
      <Paragraph position="4"> The meaning selected by PageRank from a set of possible meanings for a given word can be seen as the one most recommended by related meanings in the text, with preference given to the &amp;quot;recommendations&amp;quot; made by most influential ones, i.e. the ones that are in turn highly recommended by other related meanings. The underlying hypothesis is that in a cohesive text fragment, related meanings tend to occur together and form a &amp;quot;Web&amp;quot; of semantic connections that approximates the model humans build about a given context in the process of discourse understanding. null</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>