<?xml version="1.0" standalone="yes"?>
<Paper uid="H93-1012">
  <Title>OVERVIEW OF TREC-1</Title>
  <Section position="3" start_page="0" end_page="61" type="intro">
    <SectionTitle>
1. INTRODUCTION
</SectionTitle>
    <Paragraph position="0"> There is a long history of experimentation in information retrieval. Research started with experiments in indexing languages, such as the Cranfield I tests \[1\], and has continued with over 30 years of experimentation with the retrieval engines themselves. The Cranfield II studies \[2\] showed that automatic indexing was comparable to manual indexing, and this and the availability of computers created a major interest in the automatic indexing and searching of texts. The Cranfield experiments also emphasized the importance of creating test collections and using these for comparative evaluation.</Paragraph>
    <Paragraph position="1"> The Cranfield collection, created in the late 1960's, contained 1400 documents and 225 queries, and has been heavily used by researchers since then. Subsequently other collections have been built, such as the CACM collection \[3\], and the NPL collection \[4\].</Paragraph>
    <Paragraph position="2"> In the thirty or so years of experimentation there have been two missing elements. First, although some research groups have used the same collections, there has been no concerted effort by groups to work with the same data, use the same evaluation techniques, and generally compare results across systems. The importance of this is not to show any system to be superior, but to allow comparison across a very wide variety of techniques, much wider than only one research group would tackle. Karen Sparck Jones in 1981 \[5\] commented that: Yet the most slriking feature of the test history of the past two decades is its lack of consolidation. It is true that some very broad generalizations have been endorsed by successive tests: for example...but there has been a real failure at the detailed level to build one test on another. As a result there are no explanations for these generalizations, and hence no means of knowing whether improved systems could be designed (p. 245).</Paragraph>
    <Paragraph position="3"> This consolidation is more likely ff groups can compare results across the same data, using the same evaluation method, and then meet to discuss openly how methods differ.</Paragraph>
    <Paragraph position="4"> The second missing element, which has become critical in the last ten years, is the lack of a realisticallysized test collection. Evaluation using the small collections currently available may not reflect performance of systems in large full-text searching, and certainly does not demonstrate any proven abilities of these systems to operate in real-world information retrieval environments.</Paragraph>
    <Paragraph position="5"> This is a major barrier to the transfer of these laboratory systems into the commercial world. Additionally some techniques such as the use of phrases and the construction of automatic thesaurii seem intuitively workable, but have repeatedly failed to show improvement in performance using the small collections. Larger collections might demonslrate the effectiveness of these procedures.</Paragraph>
    <Paragraph position="6"> The overall goal of the Text REtrieval Conference (TREC) is to address these two missing elements. It is hoped that by providing a very large test collection, and encouraging interaction with other groups in a friendly evaluation forum, a new thrust in information retrieval will occur. There is also an increased interest in this field within the DARPA community, and TREC is designed to be a showcase of the state-of-the-art in retrieval research. NIST's goal as co-sponsor of TREC is to encourage communication and technology transfer among academia, industry, and government.</Paragraph>
    <Paragraph position="7"> The following description was excerpted from a more lengthy overview published in the conference proceedings \[6\]. The full proceedings also contain papers by all participants and results for all systems.</Paragraph>
  </Section>
class="xml-element"></Paper>