<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-0815">
  <Title>Dependency Based Logical Form Transformations</Title>
  <Section position="4" start_page="0" end_page="0" type="metho">
    <SectionTitle>
3 Evaluation
</SectionTitle>
    <Paragraph position="0"> Argument, predicate, and sentence level precision and recall measures are used to evaluate performance of the system as compared to a gold-standard. The system was trained on a set of 50 sentences with corresponding logical forms. Final testing was performed on a set of 300 LF -sentence pairs.</Paragraph>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
3.1 Argument Level
</SectionTitle>
      <Paragraph position="0"> Precision at the argument level is defined to be the number of correctly identified arguments divided by the number of all identified arguments. Recall is defined to be the number of correctly identified arguments divided by the real number of arguments that should be present in the target transformation.</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
3.2 Predicate Level
</SectionTitle>
      <Paragraph position="0"> Predicates must identify all arguments correctly to be counted as a correct predicate. Precision is defined to be the number of correctly ident ified predicates divided by the number of all attempted predicates. Recall is defined as the number of correctly identified predicates divided by the real number of predicates that were supposed to be identified in the target transformation.</Paragraph>
    </Section>
    <Section position="3" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
3.3 Sentence Level
</SectionTitle>
      <Paragraph position="0"> Various oth er sentence level measures are also used. Sentence-argument is defined as the number of sentences that have all arguments correctly identified divided by the number of sentences attempted.</Paragraph>
      <Paragraph position="1"> Sentence-predicate is similar except conditioned on predicates. Sentence-argument-predicate is defined to be the number of sentences that have all arguments correctly identified divided by the number of sentences which have all predicates correctly identified. Sentence-argument-predicate-sentences refers to the number of sentences that have all arguments and all pred icates correctly identified divided by the number of sentences attempted.</Paragraph>
    </Section>
  </Section>
  <Section position="5" start_page="0" end_page="0" type="metho">
    <SectionTitle>
4 Results
</SectionTitle>
    <Paragraph position="0"> As stated earlier the final evaluation was conducted on a set of 300 sentence-LF pairs. Table 4 lists the evaluation precision and recall results using the measures discussed in section 3 which have been converted into percentages.</Paragraph>
    <Paragraph position="1">  The major source of error in terms of arguments originated from the parser's inappropriate ha ndling of coordinating conjunctions. Another common source of error arose from poor handling of nominal group complexes. With regard to predicate performance, the decision to forfeit the use of the available multi-word item list proved costly.</Paragraph>
  </Section>
  <Section position="6" start_page="0" end_page="0" type="metho">
    <SectionTitle>
5 Future Work
</SectionTitle>
    <Paragraph position="0"> Harabagiu et al. (1999) proposed a scheme for attaching sense tags to predicates within the framework of transforming WordNet glosses into a logical form. In this way conceptual predicates may be formed to manipulate a meaning representation in more significant ways. Naturally the sense inventory must be sensitive enough to allow for a meanin gful and representative mutation to be applied to the meaning representation.</Paragraph>
  </Section>
class="xml-element"></Paper>