<?xml version="1.0" standalone="yes"?>
<Paper uid="X93-1007">
  <Title>DOCUMENT DETECTION SUMMARY OF RESULTS</Title>
  <Section position="2" start_page="0" end_page="0" type="intro">
    <SectionTitle>
1. INTRODUCTION
</SectionTitle>
    <Paragraph position="0"> This section presents a summary of the TIPSTER results, including some comparative system performance and some conclusions about the success of the detection half of the TIPSTER phase I project. For more details on the individual experiments, please see the system overviews.</Paragraph>
    <Paragraph position="1"> Four contractors were involved in the document detection half of TIPSTER. Two of the contractors worked in English only (Syracuse University and HNC Inc.), one contractor worked in Japanese only (TRW Systems Development Division), and one contractor worked in both languages (University of Massachusetts at Amherst). The four contractors had extremely varied approaches to the detection task. TRW transformed an operational English retrieval system (based on pattern matching using a fast hardware approach), into a Japanese version of the same operation, with a special interface designed to facilitate work in Japanese. The University of Massachusetts approach involved taking a relatively small experimental system using a probabilistic inference net methodology, scaling it up to handle the very large amounts of text and long topics in TIPSTER, and modifying the algorithms to handle Japanese. Both Syracuse University and HNC Inc.</Paragraph>
    <Paragraph position="2"> built completely new systems to handle the English collection. In the case of Syracuse University, their system is based heavily on a natural language approach to retrieval, with many of the techniques traditionally used in document understanding applied to the retrieval task. HNC Inc. took a totally different approach, applying statistical techniques based on robust mathematical models (including the use of neural networks).</Paragraph>
    <Paragraph position="3"> There were three evaluations of the contractors' work; one at 12 months, one at 18 months, and the final one at 24 months. In each case, the contractors working in English have made multiple experimental runs using the test collection, and turned in the top list of documents found.</Paragraph>
    <Paragraph position="4"> These results were first used to create the sample pool for assessment, and then were scored against the correct answers based on results from all runs (including TREC-I runs for the 18-month evaluation and TREC-2 runs for the 24-month evaluation). Standard tables using recall/precision and recall/fallout measures were distributed and compared. The evaluation of the Japanese work took place only at the 24-month period.</Paragraph>
  </Section>
class="xml-element"></Paper>