<?xml version="1.0" standalone="yes"?>
<Paper uid="A88-1002">
  <Title>A News Story Categorization System</Title>
  <Section position="5" start_page="12" end_page="12" type="metho">
    <SectionTitle>
\[RAN ANNOUNCES END OF MAJOR
OFFENSIVE IN GULF WAR
</SectionTitle>
    <Paragraph position="0"> LONDON, Feb 26 - Iran announced tonight that its major offensive against Iraq in the Gulf war had ended after dealing savage blows against the Baghdad government.</Paragraph>
    <Paragraph position="1"> \[..3 The statement by the Iranian High Command appeared to herald the close of an assault on the port city of Basra in southern Iraq.</Paragraph>
    <Paragraph position="2"> \[..3 It said 81 Iraqi brigades and battalions were totally destroyed, along with 700 tanks and 1,500 other vehicles. The victory list also included 80 warplanes downed, 250 anti.aircraft guns and 400 pieces of military hardware destroyed and the seizure of 220 tanks and armored personnel carriers.</Paragraph>
    <Paragraph position="3"> For the story that follows, the topics war and disorders are also originally hypothesized. In the confirmation phase, two things are discovered: the story mentions no wars by name nor contains any references to countries or organizations involved in conflicts that are classified as wars; and there is nothing in the story that suggests that the topic disorders should be disconfirrned. Hence war is disconf'm-ned and disorders is confirmed.</Paragraph>
  </Section>
  <Section position="6" start_page="12" end_page="12" type="metho">
    <SectionTitle>
RIOT REPORTED IN SOUTH KOREAN
PRISON
</SectionTitle>
    <Paragraph position="0"> Seoul, July 5 - Twelve South Korean women detainees refused food for the fifth consecutive day today after a riot against their maltreatment in a Seoul prison was put down, dissident sources said.</Paragraph>
    <Paragraph position="1"> The 12, detained for anti-government protests and awaiting trial, pushed away prison officials, smashed windows and occupied a prison building on Tuesday as a protest against what they called &amp;quot;suppression of prisoners' human rights&amp;quot;.</Paragraph>
    <Paragraph position="2"> After two hours, about 40 riot police, firing tear gas, stormed the building and overpowered the protesters, the sources said. Some protesters were injured, they added.</Paragraph>
    <Paragraph position="3"> For the story below, both war and disorders are hypothesized and then disconfirmed because tennis is matched during the disconf'u'mation phase.</Paragraph>
  </Section>
  <Section position="7" start_page="12" end_page="12" type="metho">
    <SectionTitle>
LENDL DEMONSTRATES GRASS
COURT MATURITY
</SectionTitle>
    <Paragraph position="0"> LONDON, July 2 - Czechoslovak top seed Ivan Lendl served warning that he may finally have come of age on grass when he emerged victorious from a pitched battle with one of the finest exponents of the fast court game at Wimbledon today.</Paragraph>
    <Paragraph position="1"> The U.S. and French Open tennis champion has never won a title on grass but he outlasted American 10th seed Tim Mayotte 6-4 4-6 6-4 3-6 9-7 over three and a half hours to join Boris Becker, Henri Leconte and Slobodan Zivojinovic in Friday's semifinals.</Paragraph>
    <Paragraph position="2"> The titanic struggle on court one upstaged the centre court clash between seventh seed Leconte and the remarkable Australian Pat Cash, which had been billed as the day's main attraction \[...\] The story below is the rare sports story which is also a disorders story. Even though the name of a sporting event, Asian Games, occurs in the text, the topic disorders is not disconfirmed. The reason is that the confirmation patternsets match words and phrases in the story (e.g. radicals and violent protests) that very strongly suggest that real disorders are being described.</Paragraph>
  </Section>
  <Section position="8" start_page="12" end_page="13" type="metho">
    <SectionTitle>
POLICE SEEK 160 SOUTH KOREAN
RADICALS
</SectionTitle>
    <Paragraph position="0"> SEOUL, July 2 - Police said today they wanted to detain 160 South Koreans to stop sabotage attempts during September's Asian Games in Seoul.</Paragraph>
    <Paragraph position="1"> The 160, mostly students and workers, masterminded various violent protests against the government and the United States in the past months but managed to escape arrest, police said.</Paragraph>
    <Paragraph position="2"> They had been tipped that the radicals were trying to organise big demonstrations against the government during the Asiad, which is to run from September 20 to October 5.</Paragraph>
    <Paragraph position="3"> &amp;quot;It is highly probable that they will form radical underground groups to step up their anti-government and anti-U.S, protests and may disrupt the Asian Games in an attempt to defame the government,&amp;quot; a senior police officer told reporters.</Paragraph>
    <Paragraph position="4"> \[...\]</Paragraph>
    <Section position="1" start_page="12" end_page="13" type="sub_section">
      <SectionTitle>
3.4. Flow of Control
</SectionTitle>
      <Paragraph position="0"> Rather than being expressed in a formal rule language, topic hypothesization and confirmation rules are specified through a lisp program. Having a  program allows for fine-grained control by the rule developer. Rather than having a set of hypothesization and conf'u'mation rules which are processed in a fixed order, we allow the rule developer to specify the order and manner of processing in a topic-dependent manner. The major kinds of activities available to rule developers for incorporation into the control code are the following: running one or more patternsets, applying evaluation functions to the resulting matches, and confirming or disconfirming topics.</Paragraph>
      <Paragraph position="1"> In developing the system, we observed many regularities in the lisp code which controls the flow of processing and we believe it would be possible and profitable to provide rule developers with a more restricted control language which embodies many of these regularities in its primitives.</Paragraph>
    </Section>
    <Section position="2" start_page="13" end_page="13" type="sub_section">
      <SectionTitle>
3.5. Rulebase Development
</SectionTitle>
      <Paragraph position="0"> The process of formulating the rulebase of the system, i.e. the collection of patterns, patternsets, and hypothesization and confirmation rules it uses, is an empirical one. It requires human rule developers to examine many stories, create rulebase components according to their intuitions, run stories through the system, observe the results, and modify the system to avoid any miscategorizations that have occurred without introducing new miscategorizations. This task is time-consuming and sometimes tedious.</Paragraph>
      <Paragraph position="1"> Nevertheless, our experience with the system suggests that it does tend to converge without undue oscillation at an accuracy level that while far from perfect is adequate for many tasks of practical importance (see Section 4). The rule development effort on this system took approximately six person months.</Paragraph>
      <Paragraph position="2"> An important factor in the success of the rulebase development effort was the separation of the vocabulary the system looks for into a collection of abstract concepts represented by patternsets. The patternsets provide rule developers with a way of thinking about the themes they are looking for in a story when they write the hypothesization and confh'mation rules without becoming mired in questions about which specific words and phrases indicate those themes.</Paragraph>
      <Paragraph position="3"> In designing the system, we also considered a different approach in which the selection of words and phrases to look for would be determined automatically by a statistical method. Since we did not adopt this approach, we have no direct evidence that it would not have worked as well as the labor-intensive method chosen. However, our choice was influenced by a belief that a statistical method would not provide us with a choice of words and phrases that could be used to make distinctions as precisely as the patterns of the kind described above that were chosen by humans.</Paragraph>
      <Paragraph position="4"> As shown in \[2\], accuracy is particularly problematic with a traditional keyword approach regardless of whether the keywords are selected by humans or statistically. And if we had adopted a statistical approach, it would have been computationally expensive to vary the length of the phrases chosen as much as human rule developers do.</Paragraph>
      <Paragraph position="5"> It would also have been difficult to establish the contextual restrictions that human rule developers establish (e.g. this word, so long as it is not followed by one of these four others). Rules of the complexity of the confirmation rule for war and disorders described in Section 3.3 are of course essentially impossible to establish by statistical means.</Paragraph>
      <Paragraph position="6"> Some interesting possibilities for a statistical approach to defining keywords have appeared recently in conjunction with semantic information about potential keywords \[7\] and in conjunction with very powerful parallel hardware devices \[4\].</Paragraph>
      <Paragraph position="7"> However, given the current state of the art, we continue to believe that our decision to use rules formulated and refined by human developers was a sound one from the point of view of the accuracy of the resulting system.</Paragraph>
    </Section>
  </Section>
  <Section position="9" start_page="13" end_page="14" type="metho">
    <SectionTitle>
4. Performance
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="13" end_page="14" type="sub_section">
      <SectionTitle>
4.1. Measuring Performance
</SectionTitle>
      <Paragraph position="0"> The accuracy of the system for topic assignments was measured through two percentages for each of the six topics: * recall: the percentage of stories assigned the topic code by human categorizers that were also assigned that code by the system; * precision: the percentage of stories assigned the topic code by the system that actually carried the topic code assigned by the human categorizers.</Paragraph>
      <Paragraph position="1">  The recall rate serves as a measure of the number of stories for which the system misses an appropriate topic code; a high recall percentage will therefore mean few such false negatives. The precision rate, on the other hand, measures the number of stories for which the system chooses an incorrect topic. A high precision percentage means few such false positives. We emphasized high recall over high precision.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>