<?xml version="1.0" standalone="yes"?>
<Paper uid="T75-2025">
  <Title>Step I: Select a phenomenon: CONTRADICTION Step 2: Select an input form: ENGLISH SENTENCES Step 3: Select an output form: ENGLISH SENTENCES THAT CONTRADICT THE INPUT SENTENCES</Title>
  <Section position="3" start_page="126" end_page="126" type="metho">
    <SectionTitle>
PRESENT ADVANTAGES
</SectionTitle>
    <Paragraph position="0"> Of the great diversity of approaches to language, the process approach represented at the workshop is uniquely capable. The two key methodological problems in the study of language over the last 2,500 years or so have been the problem of rigor and the problem of complexity. The problem of rigor in the use of natural language led to formal logics and to Godel. The problem of complexity has led to various strong reductions on the general phenomena, with tools Such as the Osgood Semantic Differential, or paired-associate tests.</Paragraph>
    <Paragraph position="1"> Sequential-order phenomena and individual use of language tend to get badly obscured.</Paragraph>
    <Paragraph position="2"> Process theory approaches the problem of rigor with methods by which process specifications are made very explicit. It approaches the problem of complexity with computers, that can hold and make use of very large numbers of processes at once.</Paragraph>
    <Paragraph position="3"> The compatability and effective coverage of large collections of hypotheses can now actually be tested.</Paragraph>
    <Paragraph position="4"> * These are exciting, reorienting advantages that make me prefer the process approach to any other, to hold high hopes for its success, and to want it to be built on good foundations.</Paragraph>
  </Section>
  <Section position="4" start_page="126" end_page="126" type="metho">
    <SectionTitle>
WHAT MAKES A DIFFERENCE?
</SectionTitle>
    <Paragraph position="0"> What do we want out of our methodology? Three characteristics of a methodology are particularly important: reliability efficiency integrative power Reliability encompasses all of those things that make experiments trustworthy at face value, including repeatability, clarity</Paragraph>
    <Paragraph position="2"> of definition and freedom from various kinds of circumstantial effects that might be responsible for success. Efficiency addresses the effort required to achieve particular results. (You don't plan to do basic genetics studies on elephants; you may prefer fruit-flies as subjects.) It deals not only with the costs of performing the work, but with support costs as well.</Paragraph>
    <Paragraph position="3"> Integrative power involves the scope of the theories, what diversity of phenomena they cover, what subtheories they coordinate, what kinds of investigations they facilitate.</Paragraph>
    <Paragraph position="4"> In order to discuss current practices we need some representative example. The one here is deliberately simple and not identified with a particular development effort. However it is composed of elements that seem to be widely used.</Paragraph>
  </Section>
  <Section position="5" start_page="126" end_page="126" type="metho">
    <SectionTitle>
SOME STRENGTHS IN CURRENT PRACTICE
</SectionTitle>
    <Paragraph position="0"> We should hold on to the distinctive strengths of our methods in any changes we plan. These strengths are generally direct classic consequences of the use of computers to hold models: Complexity of data and theory is easy to accommodate.</Paragraph>
    <Paragraph position="1"> Time sequences and dependencies are preserved.</Paragraph>
    <Paragraph position="2"> A diversity of hypotheses can be applied and tested for consistency in each experiment.</Paragraph>
    <Paragraph position="3"> All of these have to do with integrative power, and on this dimension we are, at least potentially, in very good shape.</Paragraph>
  </Section>
  <Section position="6" start_page="126" end_page="127" type="metho">
    <SectionTitle>
SOME WEAKNESSES
</SectionTitle>
    <Paragraph position="0"> We have some serious problems. Here are some recurring problems with the FORM of the work:  I. Single experiments often take years to execute.</Paragraph>
    <Paragraph position="1"> 2. The activity is often treated  as programming and program documentation rather than science. The consequences are generally that the data are poorly identified and poorly chosen, the status of the programs as theory is not clear, the business of making clear theoretical claims is neglected, and the relevance of the activity to existing theories that are not programs is never established. The remainder of science is thus cut off, and left wondering whether we are into science at all.</Paragraph>
    <Paragraph position="2"> 3. The attempt to perform a general transaction, such as Sentence:Contradiction, strongly limits the complexity of the input that gets actually addressed, with the result that significant phenomena are missed. The effects of prior context, speakers&amp;quot; goals, tacit mutual knowledge of speaker and hearer are often attenuated by the attempt to be general.</Paragraph>
    <Paragraph position="3"> 4. The unit of production is a system. Whole systems are difficult to disseminate and difficult to judge as scientific hypotheses, and are not generally understood or appreciated by non-programming scientists.</Paragraph>
    <Paragraph position="4"> 5. Coping with ad-hocness is a problem: The system runs the examples, but what else it will do is unclear, or, the degree of tuning to the examples is unclear, or, the representativeness of the examples is unclear, or, the rightness of the answers is only established intuitively.</Paragraph>
    <Paragraph position="5"> We have problems with the CONTENT of the work. There are many problems, which may be a healthy condition, but I want to attend to just one that seems to be otherwise.</Paragraph>
    <Paragraph position="6"> In the common notion, a natural language is a scheme of communication that people use. The fact that a language is used to communicate has strong consequences. For example, as languages change, their adequacy for communication must be maintained. The communication properties of language are being ignored in a wide variety of approaches, including processing approaches. Often, it is outside of the paradigmatic scope of the studies.</Paragraph>
    <Paragraph position="7"> Communication deals with changing correspondences between the knowledge of one individual or system and the knowledge of another. It is more than relations between strings and strings, or relations between strings and generators of strings (syntax). It is more than relations between strings and a world or a data base (semantics). Communication involves two active processors, and an adequate theory of language will specify some consequences of that fact. By restricting the view to a single processor (or less), I suspect that we are cutting ourselves off from the organizing principles that produce the regularities that we are trying to study. Some of the changes of style that I would suggest are implicit in ~he identifications of the problems cited above: Design clear data collection methods.</Paragraph>
    <Paragraph position="8"> State theoretical claims that are distinct from the programs.</Paragraph>
    <Paragraph position="9"> (The claims may still contain algorithms, of course.) Decommit from attempts to be general, except where an empirical demonstration of generality is included in the work.</Paragraph>
    <Paragraph position="10"> Shift from focus on systems to focus on algorithms.</Paragraph>
    <Paragraph position="11"> Do something to drastically shorten the period required to do single experiments.</Paragraph>
    <Paragraph position="12"> Beyond these suggestions, the special advantages of case analysis should be considered.</Paragraph>
  </Section>
  <Section position="7" start_page="127" end_page="127" type="metho">
    <SectionTitle>
CASE ANALYSIS AS THE BASIS FOR AN ALTERNATE
PROCESSING METHODOLOGY
</SectionTitle>
    <Paragraph position="0"> Case analysis as a basic scientific activity is an attractive alternative to the current methodology sketched above. How would it work?</Paragraph>
  </Section>
  <Section position="8" start_page="127" end_page="128" type="metho">
    <SectionTitle>
STEPS IN A CASE-ANALYSIS-BASED DEVELOPMENT
IN
NATURAL LANGUAGE PROCESSING
</SectionTitle>
    <Paragraph position="0"> IDENTIFICATION: The data are annotated and scored for particular phenomena of interest. Data can be scored for several phenomena at once. Scoring is performed by people who understand the language and the circumstances of the data occurrence, and who are given explicit instructions on what to look for and how to annotate it.</Paragraph>
    <Paragraph position="1"> The result of this step is a Commentary on the data.</Paragraph>
    <Paragraph position="2"> Examples: a. Identify requests and judge whether they are fulfilled in running dialogue.</Paragraph>
    <Paragraph position="3">  Custom-build for this data, a new one-shot program that will take the data as input, and make entries into a simulated Hearer's Memory. The program is the Model, and its &amp;quot;output&amp;quot; is its trace. Step 4: MODEL EVALUATION: Compare the Commentary with the execution trace of the model. For each significant event identified in the Commentary, decide whether there was a correctly corresponding event in the model's execution.</Paragraph>
    <Paragraph position="4"> With suitable selections of phenomena for study, it is not hard to decide whether the program performed appropriately. However, a serious problem remains: a program for a single case can be entirely ad hoc. This is an advantage, in that it is certain beforehand that the program will run successfully, independent of the complexity of the phenomena. But the program may or may not have any long-term significance. The program is composed of cooperating processes. Each process can be considered to be an over-specified hypothesis, over-specified because details such as the programming language are inessential to the corresponding functional claims about language.</Paragraph>
    <Paragraph position="5"> VERIFICATION STEP: In order to meet the ad-hocness problem, these hypotheses must be verified by repeated application to a diversity of cases. The experiment steps cited above must be repeated, and their results compared.</Paragraph>
    <Paragraph position="6"> Inessential details (such as programming language and machine) may be changed, if desired, but the properties of the algorithms which form the basis for the theoretical claims of the work must be held constant.</Paragraph>
    <Paragraph position="7"> The verified results are those algorithms that continue to work correctly, when their actions are judged against the Commentary, in model after model. These algorithms are the valuable ones both for practical application and for scientific knowledge.</Paragraph>
  </Section>
  <Section position="9" start_page="128" end_page="128" type="metho">
    <SectionTitle>
ADVANTAGES OF CASE ANALYSIS
METHODOLOGY
</SectionTitle>
    <Paragraph position="0"> Since the data acquisition step is first rather than nearly last, stronger claims can be made for the ability to model real-world phenomena. Having the data in hand is a strong guide to implementation.</Paragraph>
    <Paragraph position="1"> Because phenomena identification is explicit, and proceeds from explicit instructions, the resulting theory has a  clear operational interpretation, since it substitutes powerful hindsight for less-powerful anticipation.</Paragraph>
    <Paragraph position="2"> There is better control on complexity and effort, since no claims are made for the generality of the whole systems that are built. The amount of data modeled can be controlled, and a diversity of data sources can be accommodated. There is strong control over the involvement of world-knowledge in models, since most of the particulars can be anticipated by looking at the data.</Paragraph>
    <Paragraph position="3"> The method can also be controlled by choices about whether several phenomena will be modeled in a single model or several smaller models. The smaller models are simpler, but the single model exhibits the compatability of the parts and the consistency of the set of hypotheses.</Paragraph>
    <Paragraph position="4"> This approach typically runs in a more data-driven, phenomena-responsive manner than a general system building approach. It avoids the situation in which system design is based on inadequate stereotypes of what might happen at the input. Programming can be more goal-directed as well, since the phenomena of interest have already been identified in the Commentary.</Paragraph>
    <Paragraph position="5"> The problems of ad-hocness are treated explicitly, rather than being left to the suspicions of the journal readers. This facilitates representations of the degree and kinds of tests that the theories have had. (I suspect that for some current systems, many readers believe that they will only run the explanatory examples in the papers).</Paragraph>
    <Paragraph position="6"> Finally, because of the close control and 20-20 hindsight of case analysis, more complex phenomena can be accommodated. In particular, communication between two non-identical human processors can be modeled.</Paragraph>
  </Section>
  <Section position="10" start_page="128" end_page="129" type="metho">
    <SectionTitle>
AN ACTIVE EXAMPLE OF CASE MODELING
METHODOLOGY
</SectionTitle>
    <Paragraph position="0"> The Dialogue Process Modeling work at ISI is an active attempt to apply the ideas above with some embellishments, to real natural language processing problems. All of the recommendations are being used in identifiable ways. This work will be described in discussion at the conference as time permits.</Paragraph>
  </Section>
class="xml-element"></Paper>