<?xml version="1.0" standalone="yes"?>
<Paper uid="N04-1013">
  <Title>Speed and Accuracy in Shallow and Deep Stochastic Parsing</Title>
  <Section position="2" start_page="0" end_page="0" type="intro">
    <SectionTitle>
1 Introduction
</SectionTitle>
    <Paragraph position="0"> In applications that are sensitive to the meanings expressed by natural language sentences, it has become common in recent years simply to incorporate publicly available statistical parsers. A state-of-the-art statistical parsing system that enjoys great popularity in research systems is the parser described in Collins (1999) (henceforth &amp;quot;the Collins parser&amp;quot;). This system not only is frequently used for off-line data preprocessing, but also is included as a black-box component for applications such as document summarization (Daume and Marcu, 2002), information extraction (Miller et al., 2000), machine translation (Yamada and Knight, 2001), and question answering (Harabagiu et al., 2001). This is be- null program. We would like to thank Chris Culy whose original experiments inspired this research.</Paragraph>
    <Paragraph position="1"> cause the Collins parser shares the property of robustness with other statistical parsers, but more than other such parsers, the categories of its parse-trees make grammatical distinctions that presumably are useful for meaning-sensitive applications. For example, the categories of the Model 3 Collins parser distinguish between heads, arguments, and adjuncts and they mark some long-distance dependency paths; these distinctions can guide application-specific postprocessors in extracting important semantic relations.</Paragraph>
    <Paragraph position="2"> In contrast, state-of-the-art parsing systems based on deep grammars mark explicitly and in much more detail a wider variety of syntactic and semantic dependencies and should therefore provide even better support for meaning-sensitive applications. But common wisdom has it that parsing systems based on deep linguistic grammars are too difficult to produce, lack coverage and robustness, and also have poor run-time performance. The Collins parser is thought to be accurate and fast and thus to represent a reasonable trade-off between &amp;quot;good-enough&amp;quot; output, speed, and robustness.</Paragraph>
    <Paragraph position="3"> This paper reports on some experiments that put this conventional wisdom to an empirical test. We investigated the accuracy of recovering semantically-relevant grammatical dependencies from the tree-structures produced by the Collins parser, comparing these dependencies to gold-standard dependencies which are available for a subset of 700 sentences randomly drawn from section 23 of the Wall Street Journal (see King et al. (2003)). We compared the output of the XLE system, a deep-grammar-based parsing system using the English Lexical-Functional Grammar previously constructed as part of the Pargram project (Butt et al., 2002), to the same gold standard. This system incorporates sophisticated ambiguity-management technology so that all possible syntactic analyses of a sentence are computed in an efficient, packed representation (Maxwell and Kaplan, 1993). In accordance with LFG theory, the output includes not only standard context-free phrase-structure trees but also attribute-value matrices (LFG's f(unctional) structures) that explicitly encode predicate-argument relations and other meaningful properties. XLE selects the most probable analysis from the potentially large candidate set by means of a stochastic disambiguation component based on a log-linear (a.k.a. maximum-entropy) probability model (Riezler et al., 2002). The stochastic component is also &amp;quot;ambiguity-enabled&amp;quot; in the sense that the computations for statistical estimation and selection of the most probable analyses are done efficiently by dynamic programming, avoiding the need to unpack the parse forests and enumerate individual analyses. The underlying parsing system also has built-in robustness mechanisms that allow it to parse strings that are outside the scope of the grammar as a shortest sequence of well-formed &amp;quot;fragments&amp;quot;. Furthermore, performance parameters that bound parsing and disambiguation work can be tuned for efficient but accurate operation.</Paragraph>
    <Paragraph position="4"> As part of our assessment, we also measured the parsing speed of the two systems, taking into account all stages of processing that each system requires to produce its output. For example, since the Collins parser depends on a prior part-of-speech tagger (Ratnaparkhi, 1996), we included the time for POS tagging in our Collins measurements. XLE incorporates a sophisticated finite-state morphology and dictionary lookup component, and its time is part of the measure of XLE performance.</Paragraph>
    <Paragraph position="5"> Performance parameters of both the Collins parser and the XLE system were adjusted on a heldout set consisting of a random selection of 1/5 of the PARC 700 dependency bank; experimental results were then based on the other 560 sentences. For Model 3 of the Collins parser, a beam size of 1000, and not the recommended beam size of 10000, was found to optimize parsing speed at little loss in accuracy. On the same heldout set, parameters of the stochastic disambiguation system and parameters for parsing performance were adjusted for a Core and a Complete version of the XLE system, differing in the size of the constraint-set of the underlying grammar.</Paragraph>
    <Paragraph position="6"> For both XLE and the Collins parser we wrote conversion programs to transform the normal (tree or fstructure) output into the corresponding relations of the dependency bank. This conversion was relatively straightforward for LFG structures (King et al., 2003).</Paragraph>
    <Paragraph position="7"> However, a certain amount of skill and intuition was required to provide a fair conversion of the Collins trees: we did not want to penalize configurations in the Collins trees that encoded alternative but equally legitimate representations of the same linguistic properties (e.g. whether auxiliaries are encoded as main verbs or aspect features), but we also did not want to build into the conversion program transformations that compensate for information that Collins cannot provide without appealing to additional linguistic resources (such as identifying the subjects of infinitival complements). We did not include the time for dependency conversion in our measures of performance.</Paragraph>
    <Paragraph position="8"> The experimental results show that stochastic parsing with the Core LFG grammar achieves a better F-score than the Collins parser at a roughly comparable parsing speed. The XLE system achieves 12% reduction in error rate over the Collins parser, that is 77.6% F-score for the XLE system versus 74.6% for the Collins parser, at a cost in parsing time of a factor of 1.49.</Paragraph>
  </Section>
class="xml-element"></Paper>