<?xml version="1.0" standalone="yes"?>
<Paper uid="W99-0624">
  <Title>Lexical ambiguity and Information Retrieval revisited</Title>
  <Section position="2" start_page="0" end_page="195" type="intro">
    <SectionTitle>
1 Introduction
</SectionTitle>
    <Paragraph position="0"> A major difficulty to experiment with lexical ambiguity issues in Information Retrieval is always to differentiate the effects of the indexing and retrieval strategy being tested from the effects of tagging errors. Some examples are:  1. In (RichardSon and Smeaton, 1995), a sophisti null cated retrieval system based on conceptual similarity resultled in a decrease of IR performance. It was not possible, however, to distinguish the effects of the strategy and the effects of automatic Wordl Sense Disambiguation (WSD) errors. In (Smeaton and Quigley, 1996), a similar strategy and a combination of manual disambiguation and very short documents -image captions- pioduced, however, an improvement of IR perforinance.</Paragraph>
    <Paragraph position="1"> 2. In (Krovetz, 1997), discriminating word senses with differefit Part-Of-Speech (as annotated by the Church :POS tagger) also harmed retrieval efficiency. Krovetz noted than more than half of the words in a dictionary that differ in POS are related i n meaning, but he could not decide whether the decrease of performance was due to the loss of such semantic relatedness or to automatic POS tagging errors.</Paragraph>
    <Paragraph position="2"> 3. In (Sanderson, 1994), the problem of discerning the effects of differentiating word senses from the effects of inaccurate disambiguation was overcome using artificially created pseudo-words (substituting, for instance, all occurrences of banana or kalashnikov for banana/kalashnikov) that could be disambiguated with 100% accuracy (substituting banana/kalashnikov back to the original term in each occurrence, either banana or kalashnikov).</Paragraph>
    <Paragraph position="3"> He found that IR processes were quite resistant to increasing degrees of lexical ambiguity, and that disambiguation harmed IR efficiency if performed with less that 90% accuracy. The question is whether real ambiguous words would behave as pseudo-words.</Paragraph>
    <Paragraph position="4"> 4. In (Schiitze and Pedersen, 1995) it was shown that sense discriminations extracted from the test collections may enhance text retrieval.</Paragraph>
    <Paragraph position="5"> However, the static sense inventories in dictionaries or thesauri -such as WordNet- have not been used satisfactorily in IR. For instance, in (Voorhees, 1994), manual expansion of TREC queries with semantically related words from WordNet only produced slight improvements with the shortest queries.</Paragraph>
    <Paragraph position="6"> In order to deal with these problems, we designed an IR test collection which is hand annotated with Part-Of-Speech and semantic tags from WordNet 1.5. This collection was first introduced in (Gonzalo et al., 1998) and it is described in Section 2. This collection is quite small for current IR standards (it is only slightly bigger than the TIME collection), but offers a unique chance to analyze the behavior of semantic approaches to IR before scaling them up to TREC-size collections (where manual tagging is unfeasible).</Paragraph>
    <Paragraph position="7"> In (Gonzalo et al., 1998), we used the manual annotations in the IR-Semcor collection to show that indexing with WordNet synsets can give significant improvements to Text Retrieval, even for large queries. Such strategy works better than the synonymy expansion in (Voorhees, 1994), probably because it identifies synonym terms but, at the same  time, it differentiates word senses.</Paragraph>
    <Paragraph position="8"> In this paper we use a variant of the IR-Semcor collection to revise the results of the experiments by Sanderson (Sanderson, 1994) and Krovetz (Krovetz, 1997) cited above. The first one is reproduced using both ambiguous pseudo-words and real ambiguous words, and the qualitative results compared. This permits us to know if our results are compatible with Sanderson experiments or not. The effect of lexical ambiguity on IR processes is discussed in Section 3, and the sensitivity of recall/precision to Word Sense Disambiguation errors in Section 4. Then, the experiment by Krovetz is reproduced with automatic and manually produced POS annotations in Section 5, in order to discern the effect of annotating POS from the effect of erroneous annotations. Finally, the richness of multiwords in WordNet 1.5 and of phrase annotations in the IR-Semcor collection are exploited in Section 6 to test whether phrases are good indexing terms or not.</Paragraph>
  </Section>
class="xml-element"></Paper>