<?xml version="1.0" standalone="yes"?>
<Paper uid="C04-1055">
  <Title>Skeletons in the parser: Using a shallow parser to improve deep parsing</Title>
  <Section position="5" start_page="3" end_page="3" type="metho">
    <SectionTitle>
3 Collins Parser Evaluation
</SectionTitle>
    <Paragraph position="0"> As a pilot experiment, we evaluated the performance of the Collins parser on a single dialog of 167 sentences from the Monroe corpus, dialog 3.</Paragraph>
    <Paragraph position="1"> We extracted context-free grammar backbones from our TRIPS gold standard parses to score the Collins' output against. The evaluation was complicated by difference in tree formats, illustrated in Figure 3.</Paragraph>
    <Paragraph position="2"> The two parsers use a different (though closely related) set of syntactic categories. The TRIPS structure generally has more levels of structure (roughly corresponding to levels in X-bar theory) than the Penn Treebank analyses (Marcus et al., 1993), in particular for base noun phrases.</Paragraph>
    <Paragraph position="3"> We converted the TRIPS category labels to their nearest equivalent in Penn Treebank inventory before scoring the Collins parser in terms of labeled precision and recall of constituents, the standard measures in the statistical parsing community. Overall recall was 32%, while precision was 64%.</Paragraph>
    <Paragraph position="4"> While we expect the Collins parser to have low recall (it generates fewer constituents overall), the low precision indicates that simply relabeling constituents on a one-for-one basis is not sufficient to resolve the differences in the two formalisms. Precision and recall broken down by constituent type is shown in Table 1.</Paragraph>
    <Paragraph position="5">  However, 82% of the sentences have no crossing brackets in the Collins parse. That is, while the parser may not generate the same set of constituents, it generates very few constituents that straddle the boundaries of any constituent in the TRIPS parse.</Paragraph>
    <Paragraph position="6"> At this level, the parsers agree about the structure of the sentences to a degree that is perhaps surprising given the very different domain on which the Collins parser is trained. This indicates that the low performance on the other measures has more to do with differences in the annotation style than real mistakes by the Collins parser.</Paragraph>
    <Paragraph position="7"> The high level of agreement on unlabeled bracketings led us to believe that the Collins structure could be used as a filter for constituents generated by the TRIPS parser. We tested this strategy in experiments reported in the following section.</Paragraph>
  </Section>
  <Section position="6" start_page="3" end_page="5" type="metho">
    <SectionTitle>
4 Experiments
</SectionTitle>
    <Paragraph position="0"> In all the experiments, we used a subset of five dialogs (consisting of 1326 utterances) from the Monroe corpus, described in 2.1. Pilot trials were conducted on dialog 3 (167 utterances), and the experiments were run with the remaining dialogs (1, 2, 4 and 5).</Paragraph>
    <Section position="1" start_page="3" end_page="5" type="sub_section">
      <SectionTitle>
4.1 Method
</SectionTitle>
      <Paragraph position="0"> The first experiment evaluates whether we can extract information from the Collins output that is reliable enough to provide significant improvements to the TRIPS parser. In order to compare our performance with (Frank et al., 2002), the test only uses utterances for which we have a gold-standard. In addition, we report our experiments only on utterances 6 words or longer (with an average of 10.3 words per utterance), as shorter utterances pose little problem for the TRIPS parser and thus running the Collins pre-processing step would not be productive. null We parsed dialogs 1, 2, 4 and 5 with the Collins parser, and extracted the phrase-level bracketing for the most reliable constituents (those which has a precision of at least 60%) in our pilot study: NP, VP and ADVP.</Paragraph>
      <Paragraph position="1">  From this information we constructed a parse skeleton for each utterance, such as the one shown in Figure 4.</Paragraph>
      <Paragraph position="2"> For our experiments we modified the TRIPS parser so that when a constituent is to be added to the chart, if the constituent type and its start and end positions are found in the skeleton then the ranking for that constituent is boosted by a small amount. In pilot trials we determined the optimal boost weight to be 3% (see Table 2).</Paragraph>
      <Paragraph position="3"> With a broad coverage grammar, it is possible that the parser could run almost indefinitely on sentences that are difficult to parse. Thus we set an upper limit on the number of constituents that can be added to the chart before the parser quits. The parser runs until it finds a complete analysis or hits this upper  gold-standard TRIPS constituents were produced by Collins, precision to how many of the produced constituents matched TRIPS, and crossing brackets to the percentage of TRIPS constituents that were violated by any bracketing produced by Collins.</Paragraph>
      <Paragraph position="4"> So [NP I] [VP guess that if [NP we] [VP send [NP one ambulance] to [NP the airport]] [NP we] [VP can [VP get [NP more people off] [ADVP quickly]]]  factor.</Paragraph>
      <Paragraph position="5"> limit.In the first experiment, this upper limit is set at 10000 constituents. In addition, we performed the same experiments with lower upper limits to explore the question of how much of the parser time is spent on the sentences that hit the maximum chart size limit. In the second experiment we used an upper limit of 5000, and in the third we used an upper limit of 1500 (the standard value for use in our real-time dialog system to avoid long delays in responding).</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>