<?xml version="1.0" standalone="yes"?>
<Paper uid="H93-1012">
  <Title>OVERVIEW OF TREC-1</Title>
  <Section position="6" start_page="89" end_page="89" type="concl">
    <SectionTitle>
4. PRELIMINARY RESULTS
</SectionTitle>
    <Paragraph position="0"> An important element of TREC was to provide a common evaluation forum. Standard recall/precision figures were calculated for each system and the tables and graphs for the results are presented in the proceedings.</Paragraph>
    <Paragraph position="1"> The results of the TREC-1 conference can be viewed only as a preliminary baseline for what can be expected from systems working with large test collections. There are several reasons for this. First, the deadlines for results were very tight, and most groups had minimal time for experiments. Additionally groups were working blindly as to what constitutes a relevant document.</Paragraph>
    <Paragraph position="2"> There were no reliable relevance judgments for training, and the use of the structured topics was completely new.</Paragraph>
    <Paragraph position="3"> It can be expected that the results seen at the second TREC conference will be much better, and also more indicative of how well a method works.</Paragraph>
    <Paragraph position="4"> However there were some clear trends that emerged.</Paragraph>
    <Paragraph position="5"> Automatic construction of queries proved to be as effective as manual construction of queries. Figure 2 shows a comparison of four sets of results, two using automatic query construction and two using manual query construction, and it can be seen that there is relatively little difference between the results.</Paragraph>
    <Paragraph position="6">  The two automatic systems shown used basically all the terms in the topic as query terms, and relied on automatic term weighting and sophisticated ranking algorithms for performance. The manual systems also used sophisticated term weighting and algorithms, but manually selected which terms to include in a query.</Paragraph>
    <Paragraph position="7"> Several minor trends were also noticeable. Systems that worked with subdocuments, or used local term context to improve term weighting, seemed particularly successful in handling the longer documents in TREC.</Paragraph>
    <Paragraph position="8"> More systems may investigate this approach in TREC-2.</Paragraph>
    <Paragraph position="9"> Also systems that attempted to expand a topic beyond its original terms (either manually or automatically) seemed to do well, although it was often hard to properly control this expansion (particularly for automatically expanded queries). These trends may continue in TREC-2 and it is expected that clearer trends will emerge as groups have more time to work at this new task.</Paragraph>
  </Section>
class="xml-element"></Paper>