<?xml version="1.0" standalone="yes"?>
<Paper uid="A92-1026">
  <Title>Robust Processing of Real-World Natural-Language Texts</Title>
  <Section position="2" start_page="0" end_page="186" type="intro">
    <SectionTitle>
1 Introduction
</SectionTitle>
    <Paragraph position="0"> If automatic text processing is to be a useful enterprise, it. must be demonstrated that the completeness and accuracy of the information extracted is adequate for the application one has in nfind. While it is clear that certain applications require only a minimal level of competence from a system, it is also true that many applicationsrequire a very high degree of completeness and a.ccuracy in text processing, and an increase in capability in either area is a clear advantage. Therefore we adopt an extremely lfigh standard against which the performance of a text processing system should be measured: it. should recover all information that is implicitly or explicitly present in the text, and it should do so without making mistakes.</Paragraph>
    <Paragraph position="1"> Tiffs standard is far beyond the state of the art. It is an impossibly high standard for human beings, let alone machines. However, progress toward adequate text processing is best. served by setting ambitious goals. For this reason we believe that, while it may be necessary in the intermediate term to settle for results that are far short of this ultimate goal, any linguistic theory or system architecture that is adopted should not be demonstrably inconsistent with attaining this objective. However, if one is interested, as we are, in the potentially successful application of these intermediate-term systems to real problems, it is impossible to ignore the question of whether they can be made efficient enough and robust enough for application.</Paragraph>
    <Section position="1" start_page="0" end_page="186" type="sub_section">
      <SectionTitle>
1.1 The TACITUS System
</SectionTitle>
      <Paragraph position="0"> The TACITUS text processing system has been under development at SRI International for the last six years.</Paragraph>
      <Paragraph position="1"> This system has been designed as a first step toward the realization of a system with very high completeness and accuracy in its ability to extract information from text. The general philosophy underlying the design oI this system is that the system, to the maximum extent possible, should not discard any information that might be semantically or pragmatically relevant to a full, correct interpretation. The effect of this design philosophy on the system architecture is manifested in the following characteristics: * TACITUS relies on a large, comprehensive lexicon containing detailed syntactic subcategorization information for each lexieal item.</Paragraph>
      <Paragraph position="2"> . TACITUS produces a parse and semantic interpretation of each sentence using a comprehensive grammar of English in which different possible predicate-argument relations are associated with different syntactic structures.</Paragraph>
      <Paragraph position="3"> * TACITUS relies on a general abductive reasoning meclmnism to uncover the implicit assumptions necessary to explain the coherence of the explicit text. These basic design decisions do not by themselves distinguish TACITUS from a number of other natural-language processing systems. However, they are somewhat controversial given the intermediate goal of producing systems that are useful for existing applications Criticism of the overall design with respect to this goal centers on the following observations: * The syntactic structure of English is very complex and no grammar of English has been constructed that has complete coverage of the syntax one encounters in real-world texts. Much of the text thai needs to be processed will lie outside the scope ol the best grammars available, and therefore canno! be understood by a. system that relies on a complet(  syntactic analysis of each sentence as a prerequisite to other processing.</Paragraph>
      <Paragraph position="4"> * Typical sentences in newspaper articles are about 25-30 words in length. Many sentences are much longer. Processing strategies that rely on producing a complete syntactic analysis of such sentences will be faced with a combinatorially intractable task, assuming in the first place that the sentences lie within the language described by the grammar.</Paragraph>
      <Paragraph position="5"> * Any grammar that successfully accounts for the range of syntactic structures encountered in real-world texts will necessarily produce many alnbigt&gt; ous analyses of most sentences. Assuming that the system can find the possible analyses of a sentence in a reasonable period of time, it is still faced with the problem of choosing the correct one from the many competing ones.</Paragraph>
      <Paragraph position="6"> Designers of application-oriented text processing systems have adopted a number of strategies for (lea.ling with these problems. Such strategies involve deemphasizing the role of syntactic analysis (Jacobs et al., 1991), producing partial parses with stochastic or heuristic parsers (de Marcken, 1990; Weischedel et al 1991) or resorting to weaker syntactic processing methods such as conceptual or case-frame based parsing (e.g., Schank and Riesbeck, 1981) or template matching techniques (Jackson et M., 1991). A common feature shared by these weaker methods is that they ignore certain information that is present in the text, which could be extracted by a more comprehensive analysis. The information that is ignored may be irrelevant to a particular application, or relevant in only an insignificant handful of cases, and thus we cannot argue that approaches to text processing based on weak or even nonexistent syntactic and semantic analysis are doomed to failure in all cases and are not worthy of further investigation. However, it is not obvious how such methods can scale up to handle fine distinctions in attachment, scoping, and inference, although some recent attempts have been made in this direction (Cardie and Lehnert, 1991b).</Paragraph>
      <Paragraph position="7"> In the development of TACITUS, we have chosen a design philosophy that assumes that a complete and accurate analysis of the text is being undertaken. In this paper we discuss how issues of robustness are approached from this general design perspective. In particular, we demonstrate that * useful partial analyses of the text can be obtained in cases in which the text is not grammatical English, or lies outside the scope of the grammar's coverage, * substantially correct parses of sentences can be found without exploring the entire search space for each sentence, * useful pragmatic interpretations can be obtained using general reasoning methods, even in cases in which the system lacks the necessary world knowledge to resolve all of the pragmatic problems posed in a sentence, and * all of this processing can be done within acceptable bounds on computational resources.</Paragraph>
      <Paragraph position="8"> Our experience with TACITUS suggests that extension of the system's capabilities to higher levels of completeness and accuracy can be achieved through incremental modifications of the system's knowledge, lexicon and grammar, while the robust processing techniques discussed in the following sections make the system usable for intermediate term applications. We have eva.luated the success of the various techniques discussed here, and conclude fi'om this eva.hlation that TAC1TUS offers substantiatioll of our claim that a text. processing system based on principles of complete syntactic, semantic and pragmatic analysis need not. be too brittle or computationally expensive for practical applications.</Paragraph>
    </Section>
    <Section position="2" start_page="186" end_page="186" type="sub_section">
      <SectionTitle>
1.2 Evaluating the System
</SectionTitle>
      <Paragraph position="0"> SRI International participated in the recent M UC,-3 evaluation of text-understanding systems (Sundheim, 1991).</Paragraph>
      <Paragraph position="1"> The methodolpgy chosen for this evaluation was to score a system's ability to fill in slots in tenlplates s,nnmarizing the content of short (approximately 1 page) newspaper articles on Latin American terrorism. The template-filling task required identifying, among other things, the perpetrators and victims of each terrorist act described in the articles, the occupation of the victims, the typ~ of physical entity attacked or destroyed, the date, tile location, and the effect on the targets. Frequently, articles described multiple incidents, while other texts were completely irrelevant.</Paragraph>
      <Paragraph position="2"> A set of 1,300 such newspaper articles was selected on the basis of the presence of keywords in the text, and given to participants as training data. Several hundred texts from the corpus were withheld for various phases of testing. Participants were scored on their ability to fill the templates correctly. Recall and precision measures were computed as an objective performance evaluation metric. Variations in computing these metrics are possible, but intuitively understood, recall measures the percentage of correct fills a system finds (ignoring wrong and spurious answers), and precision measures the percentage of correct fills provided out of the total number of answers posited. Thus, recall measures the completeness of a system's ability to extract information from a text, while precision measures it's accuracy.</Paragraph>
      <Paragraph position="3"> The TACITUS system achieved a recall of 44% with a precision of 65% on templates for events correctly identiffed, and a recall of 25% with a precision of 48% on all templates, including spurious templates the system generated. Our precision was the highest among the participating sites; our recall was somewhere in tile middle. Although pleased with these overall results, a subsequent detailed analysis of our performance on the first 20 messages of the 100-message test set is much more illuminating for evaluating the success of the particulax robust processing strategies we have chosen. In the remainder of this paper, we discuss the impact of the robust processing methods in the Tight of this detailed analysis.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>