<?xml version="1.0" standalone="yes"?>
<Paper uid="A88-1009">
  <Title>RESPONDING TO SEMANTICALLY ILL-FORMED INPUT</Title>
  <Section position="4" start_page="0" end_page="67" type="metho">
    <SectionTitle>
2. Background
A. Relative and Absolute Ill-form-
</SectionTitle>
    <Paragraph position="0"> edness Weischedel and Sondheimer (~,Veischedel, 1983) have distinguished two types of ill-formedness: absolute ill-formedness and relative ill-formedness. Roughly speaking, an absolutely ill-formed input is one which does not conform to the syntactic and semantic constraints of the natural language or the sublanguage; a relatively ill-formed input is one which is outside the coverage of a particular natural language interface. Our concern is primarily with relative ill-formedness. For complex domains, we believe that it will be difficult to create complete semantic models, and therefore that relatively ill-formed input will be a serious problem - a problem that it will be hard for users to remedy without suitable feedback.</Paragraph>
    <Paragraph position="1"> B. Syntactic and Semantic Ill-formedness null Earlier studies have examined both syntactically and semantically ill-formed input. Among the work on syntactically ill-formed input has been EPISTLE (Miller 1981), the work of Weischedel and Sondheimer (Weischedel 1980, Kwasney 1981, and Weischedel 1983), and Carbonell and Hayes (Carbonell 1983). Some of this work has involved the relaxation of syntactic constraints; other (such  as Carbonell and Hayes) a reliance primarily on semantic structures when syntactic analysis fails. Our system has been primarily motivated by our concern about the possiblity of constructing complete semantic models, so we have focussed to date on semantic ill-formedness, but we believe that our system will have to be extended in the future to handle syntactic ill-formedness as well.</Paragraph>
    <Paragraph position="2"> C. Error Identification and Correction null For some applications, it is sufficient that the point of ill-formedness be identified, and the constraint be relaxed so that an analysis can be obtained.</Paragraph>
    <Paragraph position="3"> This was the case in Wilks' early work on &amp;quot;Preference Semantics&amp;quot;(Wilks 1975), which was used for machine translation applications. In other applications it is necessary to obtain an analysis conforming to the system's semantic model in order for further processing of the input to take place, in effect &amp;quot;correcting&amp;quot; the user's input. This is the case for data base query (our current application), for command systems (such as MURPHY (Selfridge 1986)), and for message entry systems (such as NOMAD (Granger 1983) and VOX (Meyers 1985)).</Paragraph>
    <Paragraph position="4"> D. System Organization Error correction can be provided either by making pervasive changes to a set of rules, or by providing uniform correction procedures which work with a standard (non-correcting) set of rules. In the syntactic domain, EPISTLE is an example of the former, the metarule approach (Weischedel 1983) an example of the latter. We feel that, particularly for semantic correction, it is important to take the &amp;quot;uniform procedure&amp;quot; approach, since a semantic model for a large domain will be difficult enough to build and maintain without having to take the needs of a correction mechanism into account. It is equally important to have a procedure which will operate on a system with separate syntactic and semantic components, so that we may reap the advantages of such an organization (conciseness, modularity). The NOMAD system used procedures associated with individual words and so was very hard to extend (Granger 1983, p. 195); the VOX system remedied some of these defects but used a &amp;quot;conceptual grammar&amp;quot; mixing syntactic and semantic constraints (Meyers 1985). The MURPHY system (Selfridge 1986) is most similar to our own work in terms of the approach to semantic constraint relaxation and user feedback; however, it used a syntactic representation which would be difficult to extend, and required weights in the semantic model for the correction procedure. null The ill-formedness we are considering may also be viewed as one type of violation of the intensional constraints of the data base (constraints in this case on the classes of objects which may participate in particular relations). Intensional constraints have been studied in connection with natural language query systems by several researchers, including Mays (1980) and Gal (1985). In particular, the technique that we have adopted is similar in general terms to that suggested by Mays to handle non-existent relationships.</Paragraph>
    <Paragraph position="5"> In addition to the shortcomings of some of the systems just described, we felt it important to develop and test a system in order to gain experience in the effectiveness of these correction techniques. Although (as just noted) many techniques have been described, the published reports contain vir- null tually no evaluation of the different approaches.</Paragraph>
    <Paragraph position="6"> 3. System Overview  Our feedback mechanism is being evaluated in the context of a small question-answering system with a relatively standard structure. Processing of a question begins with two stages of syntax analysis: parsing, using an augmented context-free grammar, and syntactic regularization, which converts the various types of clauses (active and passive; interrogative, imperative, and declarative; relative and reduced relative; etc.) into a canonical form. In this canonical form, each clause is represented as a list consisting of: tense, aspect, and voice markers; the verb (root form); and a list of operands, each marked by &amp;quot;subject&amp;quot;, &amp;quot;object&amp;quot;, or the governing preposition. For example, &amp;quot;John received an A in calculus.&amp;quot; would be translated to (past receive (subject John) (object A) (in calculus)) Processing continues with semantic analysis, which translates the regularized parse into an extended-predicate-calculus formula. One aspect of this translation is the determination of quantifier scope. Another aspect is the mapping of each verb and its operands (subject, objects, and modifiers) into a predicate-argument structure. The predicate calculus formula is then interpreted as a data base retrieval command. Finally, the retrieved data is formatted for the user.</Paragraph>
    <Paragraph position="7"> The translation from verb plus operands to predicate plus arguments is controlled by the  model for the domain. The domain vocabulary is organized into a set of verb, noun, adjective, and adverb semantic classes. The model is a set of patterns stated in terms of these semantic classes. Each pattern represents one combination of verb and operands which is valid (meaningful) in this domain. For example, the pattern which would match the sentence given just above is (v-receive (subject nstudent) (object ngrade) (in ncourse)) where v-receive is the class of verbs including receive, get, etc.; nstudent the class of students; ngrade the class of grades; and ncourse the class of course names. Associated with each pattern is a rule for creating the corresponding predicate-argument structure.</Paragraph>
  </Section>
  <Section position="5" start_page="67" end_page="67" type="metho">
    <SectionTitle>
4. The Diagnostic Process
</SectionTitle>
    <Paragraph position="0"> In terms of the system just described, the analysis failures we are concerned with correspond to the presence in the input of clauses which do not match any pattern in the model. The essence of our approach is quite simple: find the patterns in the model which come closest to matching the input clause, and create sentences using these patterns. Implementation of this basic idea, however, has required the development of several processing steps, which we now describe.</Paragraph>
    <Paragraph position="1"> Our first task is to identify the clauses to which we should apply our diagnostic procedure.</Paragraph>
    <Paragraph position="2"> Our first impulse might be to trigger the procedure as soon as we parse a clause which doesn't match the model. However, the process of matching clause against model serves in our system to check selectional constraints. These constraints are needed to filter out, from syntactically valid analyses, those which are semantically ill-formed.</Paragraph>
    <Paragraph position="3"> In a typical query we may have several semantically ill-foimed analyses (along with one well-formed one), and thus several occasions of failure in the matching process before we obtain the correct analysis.</Paragraph>
    <Paragraph position="4"> We must therefore wait until syntax analysis is complete and see if there is any syntactic analysis satisfying all selectional constraints. If there is no such analysis, we look for an analysis in which all but one clause satisfies the selectional constraints; if there is such an analysis, we mark the offending clause as our candidate for diagnostic processing.</Paragraph>
    <Paragraph position="5"> Next we look for patterns in the model which &amp;quot;roughly match&amp;quot; this clause. As we explained above, the regularized clause contains a verb and a set of syntactic cases with case labels and fillers; each model pattern specifies a verb class and a set of cases, with each case slot specifying a label and the semantic class of its filler. We define a distance measure between a clause and a pattern by assigning a score to each type of mismatch (clause and pattern have the same syntactic case with different semantic classes; clause and pattern include the same semantic class but in different cases; clause has case not present in pattern; etc.) and adding the scores. We then select the pattern or patterns which, according to this distance measure, are closest to the offending clause.</Paragraph>
    <Paragraph position="6"> We now must take each of these patterns and build from it a sentence or phrase the user can understand. Each pattern is in effect a syntactic case frame, with slots whose values have to be filled in. If the case corresponds to one present in the clause, we copy the value from the clause; if the case is optional, we delete it. Othewise we create a slot filler consisting of an indefinite article and a noun describing the semantic class allowed in that slot (for example, if the pattern allows members of the class of students in a slot, we would generate the filler &amp;quot;a student&amp;quot;). When all the slots have been filled, we have a structure comparable to the regularized clause structure produced by syntactic analysis.</Paragraph>
    <Paragraph position="7"> Finally each filled-in pattern must be transformed to a syntactic form parallel to that of the original offending clause. (If we don't do this if, for example, the input is a yes-no question and the feedback is a declarative sentence - the system output can be quite confusing.) We isolate the tense, voice, aspect, and other syntactic features of the original clause (this is part of the syntactic regularization process) and transfer these features to the generated structure. If the offending clause is an embedded clause in the original sentence, we save the context of the offending clause (the matrix sentence) and insert the &amp;quot;corrected&amp;quot; clause into this context. We take the resulting structure and apply a sentence generation procedure. The generation procedure, guided by the syntactic feature markers, applies &amp;quot;forward&amp;quot; transformations which eventually generate a sentence string. These sentences are presented as the system's suggestions to the user.</Paragraph>
  </Section>
  <Section position="6" start_page="67" end_page="67" type="metho">
    <SectionTitle>
5. Examples
</SectionTitle>
    <Paragraph position="0"> The system has been implemented as described above, and has been tested as part of a question-answering system for a small &amp;quot;student transcript&amp;quot;  data base. The syntactic model currently has patterns for 30 combinations of verbs and arguments. While the model has been gradually growing, it still has sufficient &amp;quot;gaps&amp;quot; to give adequate opportunity for applying the diagnostics.</Paragraph>
    <Paragraph position="1"> A few examples will serve to clarify the operation of the system. The system has models (take (subject student) (object course)) and (offer (subject school) (object course)) but no model of the form (offer (subject student) (object course)) Accordingly, if a user types Did any students offer Vll? (where Vll is the name of a course), the system will respond Sorry, I don't understand the pattern (students offer courses) and will offer the &amp;quot;suggestions&amp;quot; Did any students take Vll? and Did some school offer Vll? Prepositional phrase modifiers are analyzed by inserting a &amp;quot;be&amp;quot; and treating the result as a relative clause. For example, &amp;quot;students in Vll&amp;quot; would be expanded to &amp;quot;students \[such that\] \[students\] be in Vll&amp;quot;. If the resulting clause is not in the semantic model, the usual correction procedures are applied. As part of our policy of limiting the model for testing purposes, we did not include a pattern of the form (be (subject student) (in course)) but there is a pattern of the form (enroll (subject student) (in course)) (for sentences such as &amp;quot;Tom enrolled in Vll.&amp;quot;). Therefore if the user types List the students in Vll.</Paragraph>
    <Paragraph position="2"> the system will generate the suggestions List the students who enroll in Vll.</Paragraph>
    <Paragraph position="3"> and List the students.</Paragraph>
    <Paragraph position="4"> (the second suggestion arising by deleting the modifier).</Paragraph>
  </Section>
  <Section position="7" start_page="67" end_page="67" type="metho">
    <SectionTitle>
6. Current Status
</SectionTitle>
    <Paragraph position="0"> The system has been operational since the summer of 1986. Since that time we have been regularly testing the system on various volunteers and revising the system to improve its design and feedback.</Paragraph>
    <Paragraph position="1"> We instructed the volunteers to try to use the system to get various pieces of information, rather than setting them a fixed task, so the queries tried have varied widely among users.</Paragraph>
    <Paragraph position="2"> The experimental results indicate both the strength and weakness of the technique we have described. On the one hand, semantic pattern mismatch is not the primary cause of failure; vocabulary overshoot (using words not in the dictionary) is much more common. In a series of tests involving 375 queries (by 8 users), 199 (53%) were successful, 95 (25%) failed due to missing vocabulary, 22 (6%) failed due to semantic pattern mismatch, and 59 (16%) failed for other reasons. On the other hand, in cases of semantic pattern mismatch, the suggestions made by the system usually include an appropriate rephrasing of the query (as well as some extraneous suggestions). Of the 22 failures due to semantic pattern mismatch (in both series of tests), we judge that in 14 cases the suggestions included an appropriate rephrasing.</Paragraph>
  </Section>
class="xml-element"></Paper>