<?xml version="1.0" standalone="yes"?>
<Paper uid="P98-1100">
  <Title>Text Segmentation Using Reiteration and Collocation</Title>
  <Section position="6" start_page="615" end_page="617" type="concl">
    <SectionTitle>
5 Experiment 2: Test Subject Evaluation
</SectionTitle>
    <Paragraph position="0"> The objective of the current investigation was to determine whether all troughs coincide with a subject change. The troughs placed by the  algorithm were compared to the segmentations identified by test subjects for the same texts. Method: Twenty texts were randomly selected for test data each consisting of approximately 500 words. These texts were presented to seven test subjects who were instructed to identify the sentences at which a new subject area commenced. No restriction was placed on the number of subject changes that could be identified. Segmentation points, indicating a change of subject, were determined by the agreement of three or more test subjects (Litman and Passonneau, 1996). Adjacent segmentation points were treated as one point because it is likely that they refer to the same subject change.</Paragraph>
    <Paragraph position="1"> The troughs placed by the segmentation algorithm were compared to the segmentation points identified by the test subjects. In Experiment 1, the top five approaches investigated identified at least 40 out of 42 known subject change points. Due to that success, these five approaches were applied in this experiment. To evaluate the results, the information retrieval metrics precision and recall were used. These metrics have tended to be adopted for the assessment of text segmentation algorithms, but they do not provide a scale of correctness (Beeferman et al., 1997). The degree to which a segmentation point was 'missed' by a trough, for instance, is not considered. Allowing an error margin provides some degree of flexibility. An error margin of two sentences either side of a segmentation point was used by Hearst (1993) and Reynar (1994) allowed three sentences. In this investigation, an error margin of two sentences was considered.</Paragraph>
    <Paragraph position="2"> Results: Table 2 gives the mean values for the comparison of troughs placed by the segmentation algorithm to the segmentation points identified by the test subjects for all the texts.</Paragraph>
    <Paragraph position="3"> Discussion: The segmentation algorithm using word repetition and relation weights in combination achieved mean precision and recall rates of 0.80 and 0.69, respectively. For 9 out of the 20 texts segmented, all troughs were relevant.</Paragraph>
    <Paragraph position="4"> Therefore, many of the troughs placed by the segmentation algorithm represented valid subject  mean values for all texts relevant!relevant nonrel, prec. found found rec.  points placed by the test subjects.</Paragraph>
    <Paragraph position="5"> changes. Both word repetition in combination with collocation and all three features in combination also achieved a precision rate of 0.80 but attained a lower recall rate of 0.62. These results demonstrate that supplementing word repetition with other linguistic features can improve text segmentation. As an example, a text segmentation algorithm developed by Hearst (1994) based on word repetition alone attained inferior precision and recall rates of 0.66 and 0.61.</Paragraph>
    <Paragraph position="6"> In this investigation, recall rates tended to be lower than precision rates because the algorithm identified fewer segments (4.1 per text) than the test subjects (4.5). Each text was only 500 words in length and was related to a specific subject area. These factors limited the degree of subject change that occurred. Consequently, the test subjects tended to identify subject changes that were more subtle than the algorithm could detect.</Paragraph>
    <Paragraph position="7"> Conclusion The text segmentation algorithm developed used three linguistic features to automatically detect lexical cohesion relations across windows. The combination of features word repetition and relation weights produced the best precision and recall rates of 0.80 and 0.69. When used in  isolation, the performance of each feature was inferior to a combined approach. This fact provides evidence that different lexical relations are detected by each linguistic feature considered.</Paragraph>
    <Paragraph position="8"> Areas for improving the segmentation algorithm include incorporation of a threshold for troughs. Currently, all troughs indicate a subject change, however, minor fluctuations in scores may be discounted. Future work with this algorithm should include application to longer documents. With trough thresholding the segments identified in longer documents could detect significant subject changes. Having located the related segments in text, a method of determining the subject of each segment could be developed, for example, for information retrieval purposes.</Paragraph>
  </Section>
class="xml-element"></Paper>