<?xml version="1.0" standalone="yes"?>
<Paper uid="A00-1042">
  <Title>Evaluation of Automatically Identified Index Terms for Browsing Electronic Documents I</Title>
  <Section position="3" start_page="304" end_page="305" type="metho">
    <SectionTitle>
4. Experimental Method
</SectionTitle>
    <Paragraph position="0"> To evaluate techniques, we performed a qualitative user evaluation in which the terms identified by each method were compared for usefulness as index terms.</Paragraph>
    <Section position="1" start_page="304" end_page="304" type="sub_section">
      <SectionTitle>
4.1 Subjects
</SectionTitle>
      <Paragraph position="0"> We performed our study with librarians, publishing professionals and undergraduate and graduate students at our university. 29 subjects participated in the study: 7 librarians and publishing professionals and 22 students.</Paragraph>
    </Section>
    <Section position="2" start_page="304" end_page="305" type="sub_section">
      <SectionTitle>
4.2 Data
</SectionTitle>
      <Paragraph position="0"> For this experiment, we selected three articles from the 1990 Wall Street Journal contained in the Tipster collection of documents. The articles were about 500 words in length.</Paragraph>
      <Paragraph position="1"> To compare methods, each article was processed three times: 1) with SMART to identify stemmed keywords \[Salton 1989\]; 2) with an implementation of the TT algorithm based on \[Justeson and Katz 1995\]; and 3) with our implementation of the HS method. Output for one article is shown in Appendix A. Figure 1 shows the articles selected, their length in words and the number of index terms from each method for each article presented to the subjects.</Paragraph>
      <Paragraph position="2">  The number of TTs is much lower than the number of KWs or HSs. This presented us with a problem: on the one hand, we were concerned about preserving the integrity of the three methods, each of which has their own logic, and at the same time, we were concerned to present lists that were balanced relative to each other. Toward this end, we made several decisions about presentation of the data: 1. Threshold: So that no bias would be unintentionally introduced, we presented subjects with all terms output by each method, up to a specified cut-off poin-However, using lists of equal length for each method would have necessitated either omitting HSs and KWs or changing the definition of TTs. Therefore we made the following decisions: * For TTs, we included all identified terms; * For HSs, we included all terms whose head occurred more than once in the document;</Paragraph>
      <Paragraph position="4"> * For KWs, we included all terms in order of decreasing frequency, up to the point where we observed diminishing quality and where the number of KWs approximated the number of HSs.</Paragraph>
      <Paragraph position="5"> Order: For the KW and TT approach, order is not significant. However, for the HS approach, the grouping together of phrases with common heads is, we claim, one of the advantages of the method. We therefore alphabetized the KWs and TTs in standard left to right order and alphabetized the HSs by head, e.g., trust account precedes money market fund.</Paragraph>
      <Paragraph position="6"> Morphological expansion: The KW approach identifies stems which represent a set of one or more morphological variants of the stem. Since in some cases the stem is not an English word, we expanded each stem to include the morphological variants that actually occurred in the article. For example, for the stem reject, we listed rejected and rejecting but did not list rejects, which did not occur in the article.</Paragraph>
    </Section>
    <Section position="3" start_page="305" end_page="305" type="sub_section">
      <SectionTitle>
4.3 Presentation to subjects
</SectionTitle>
      <Paragraph position="0"> Each subject was presented with three articles.</Paragraph>
      <Paragraph position="1"> For one article, the subject received a head sorted list of HSs; for another article, the sub-ject received a list of technical terms, and for the third article, the subject saw a list of keywords. No time limit was placed on the task.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>