<?xml version="1.0" standalone="yes"?>
<Paper uid="P91-1014">
  <Title>Polynomial Time and Space Shift-Reduce Parsing of Arbitrary Context-free Grammars.*</Title>
  <Section position="2" start_page="0" end_page="106" type="intro">
    <SectionTitle>
1 Introduction
</SectionTitle>
    <Paragraph position="0"> Predictive bottom-up parsers (Earley, 1968; Earley, 1970; Graham et al., 1980) are often used for natural language processing because of their superior average performance compared to purely bottom-up parsers *We are extremely indebted to Fernando Pereira and Stuart Shleber for providing valuable technical comments during discussions about earlier versio/m of this algorithm. We are also grateful to Aravind Joehi for his support of this research. We also thank Robert Frank. All remaining errors are the author's responsibility alone. This research wa~ partially funded by ARO grant DAAL03-89-C0031PRI and DARPA grant N0001490-J-1863. null such as CKY-style parsers (Kasami, 1965; Younger, 1967). Their practical superiority is mainly obtained because of the top-down filtering accomplished by the predictive component of the parser. Compiling out as much as possible this predictive component before run-time will result in a more efficient parser so long as the worst case behavior is not deteriorated.</Paragraph>
    <Paragraph position="1"> Approaches in this direction have been investigated (Earley, 1968; Lang, 1974; Tomita, 1985; Tomita, 1987), however none of them is satisfying, either because the worst case complexity is deteriorated (worse than Earley's parser) or because the technique is not general. Furthermore, none of these approaches have been formally proven to have a behavior superior to well known parsers such as Earley's parser.</Paragraph>
    <Paragraph position="2"> Earley himself (\[1968\] pages 69-89) proposed to precompile the state sets generated by his algorithm to make it as efficient as LR(k) parsers (Knuth, 1965) when used on LR(k) grammars by precomputing all possible states sets that the parser could create. However, some context-free grammars, including most likely most natural language grammars, cannot be compiled using his technique and the problem of knowing if a grammar can be compiled with this technique is undecidable (Earley \[1968\], page 99).</Paragraph>
    <Paragraph position="3"> Lang (1974) proposed a technique for evaluating in pseudo-parallel non-deterministic push down automata. Although this technique achieves a worst case complexity of O(n3)-time with respect to the length of input, it requires that at most two symbols are popped from the stack in a single move. When the technique is used for shift-reduce parsing, this constraint requires that the context-free grammar is in Chomsky normal form (CNF). As far as the grammar size is concerned, an exponential worst case behavior is reached when used with the characteristic LR(0)  machine. 1 Tomita (1985; 1987) proposed to extend LR(0) parsers to non-deterministic context-free grammars by explicitly using a graph structured stack which represents the pseudo-parallel evaluation of the moves of a non-deterministic LR(0) push-down automaton.</Paragraph>
    <Paragraph position="4"> Tomita's encoding of the non-deterministic push-down automaton suffers from an exponential time and space worst case complexity with respect to the input length and also with respect to the grammar size (Johnson \[1989\] and also page 72 in Tomita \[1985\]). Although Tomita reports experimental data that seem to show that the parser behaves in practice better than Earley's parser (which is proven to take in the worst case O(\[G\[2n3)-time), the duplication of the same experiments shows no conclusive outcome.</Paragraph>
    <Paragraph position="5"> Modifications to Tomita's algorithm have been proposed in order to alleviate the exponential complexity with respect to the input length (Kipps, 1989) but, according to Kipps, the modified algorithm does not lead to a practical parser. Furthermore, the algorithm is doomed to behave in the worst case in exponential time with respect to the grammar size for some ambiguous grammars and inputs (Johnson, 1989). 2 So far, there is no formal proof showing that the Tomita's parser can be superior for some grammars and inputs to Earley's parser, and its worst case complexity seems to contradict the experimental data.</Paragraph>
    <Paragraph position="6"> As explained, the previous attempts to compile the predictive component are not general and achieve a worst case complexity (with respect to the grammar size and the input length) worse than standard parsers.</Paragraph>
    <Paragraph position="7"> The methodology we follow in order to compile the predictive component of Earley's parser is to define a predictive bottom-up pushdown machine equivalent to the given grammar which we drive in pseudoparallel. Following Johnson's (1989) argument, any parsing algorithm based on the LR(0) characteristic machine is doomed to behave in exponential time with respect to the grammar size for some ambiguous grammars and inputs. This is a result of the fact that the number of states of an LR(0) characteristic machine can be exponential and that there are some grammars and inputs for which an exponential number of states must be reached (See Johnson \[1989\] for examples of such grammars and inputs). One must therefore design a different pushdown machine which  cessing since in this context the input length is typically small (10-20 words) and the granunar size very large (hundreds or thousands of rules and symbols).</Paragraph>
    <Paragraph position="8"> can be driven efficiently in pseudo-parallel.</Paragraph>
    <Paragraph position="9"> We construct a non-deterministic predictive push-down machine given an arbitrary context-free grammar whose number of states is proportional to the size of the grammar. Then at run time, we efficiently drive this machine in pseudo-parallel. Even if all the states of the machine are reached for some grammars and inputs, a polynomial complexity will still be obtained since the number of states is bounded by the grammar size. We therefore introduce a shift-reduce driver for this machine in which all of the predictive component has been compiled in the finite state control of the machine. The technique makes no requirement on the form of the context-free grammar and it behaves in the worst case as well as Earley's parser (Earley, 1970). The push-down machine is built before run-time and it is encoded as parsing tables in the which the predictive behavior has been compiled out.</Paragraph>
    <Paragraph position="10"> In the worst case, the recognizer behaves in the same O(\[Gl2nS)-time and O(\[G\[n2)-space as Earley's parser. However in practice it is always superior to Earley's parser since the prediction steps have been eliminated before run-time. We show that the items produced in the chart correspond to equivalence classes on the items produced for the same input by Earley's parser. This mapping formally shows its practical superior behavior. 3 Finally, we explain how other more efficient variants of the basic parser can be obtained by determinizing portions of the basic non-deterministic push-down machine while still using the same pseudo-parallel driver.</Paragraph>
  </Section>
class="xml-element"></Paper>