<?xml version="1.0" standalone="yes"?>
<Paper uid="W06-2611">
  <Title>Towards Free-text Semantic Parsing: A Unified Framework Based on FrameNet, VerbNet and PropBank</Title>
  <Section position="6" start_page="82" end_page="83" type="evalu">
    <SectionTitle>
5 Experiments
</SectionTitle>
    <Paragraph position="0"> In the previous section we have presented the algorithm for annotating the verb predicates of FrameNet with Intersective Levin classes. In order to show the effectiveness of this annotation and of the Intersective Levin class in general we have performed several experiments.</Paragraph>
    <Paragraph position="1"> First, we trained (1) an ILC multiclassifier from FrameNet, (2) an ILC multiclassifier from PropBank and (3) a frame multiclassifier from FrameNet. We compared the results obtained when trying to classify the VerbNet class with the results obtained when classifying frame. We show that Intersective Levin classes are easier to detect than FrameNet frames.</Paragraph>
    <Paragraph position="2"> Our second set of experiments regards the automatic labeling of FrameNet semantic roles on FrameNet corpus when using as features: gold frame, gold Intersective Levin class, automatically detected frame and automatically detected Intersective Levin class. We show that in all situations in which the VerbNet class feature is used, the accuracy loss, compared to the usage of the frame feature, is negligible. We thus show that the Intersective Levin class can successfully replace the frame feature for the task of semantic role labeling.</Paragraph>
    <Paragraph position="3"> Another set of experiments regards the generalization property of the Intersective Levin class. We show the impact of this feature when very few training data is available and its evolution when adding more and more training examples. We again perform the experiments for: gold frame, gold Intersective Levin class, automatically detected frame and automatically detected Intersective Levin class.</Paragraph>
    <Paragraph position="4"> Finally, we simulate the difficulty of free text by annotating PropBank with FrameNet semantic roles. We use PropBank because it is different from FrameNet from a domain point of view. This characteristic makes PropBank a difficult test bed for semantic role models trained on FrameNet.</Paragraph>
    <Paragraph position="5"> In the following section we present the results obtained for each of the experiments mentioned above.</Paragraph>
    <Section position="1" start_page="82" end_page="82" type="sub_section">
      <SectionTitle>
5.1 Experimental setup
</SectionTitle>
      <Paragraph position="0"> The corpora available for the experiments were PropBank and FrameNet. PropBank contains about 54,900 sentences and gold parse trees. We used sections from 02 to 22 (52,172 sentences) to train the Intersective Levin class classifiers and section 23 (2,742 sentences) for testing purposes.</Paragraph>
      <Paragraph position="1"> For the experiments on FrameNet corpus we extracted 58,384 sentences from the 319 frames that contain at least one verb annotation. There are 128,339 argument instances of 454 semantic roles. Only verbs are selected to be predicates in our evaluations. Moreover, as there is no fixed split between training and testing, we randomly selected 20% of sentences for testing and 80% for training. The sentences were processed using Charniak's parser (Charniak, 2000) to generate parse trees automatically.</Paragraph>
      <Paragraph position="2"> For classification, we used the SVM-light-TK software available at http://ai-nlp.</Paragraph>
      <Paragraph position="3"> info.uniroma2.it/moschitti which encodes tree kernels in the SVM-light software (Joachims, 1999). The classification performance was evaluated using the F1 measure for the single-argument classifiers and the accuracy for the multiclassifiers.</Paragraph>
    </Section>
    <Section position="2" start_page="82" end_page="83" type="sub_section">
      <SectionTitle>
5.2 Automatic VerbNet vs. automatic Fra-
</SectionTitle>
      <Paragraph position="0"> meNet frame detection In these experiments we classify Intersective Levin classes (ILC) on PropBank (PB) and FrameNet (FN) and frame on FrameNet. For the training stage we use SVMs with Tree Kernels. The main idea of tree kernels is the modeling of a KT(T1,T2) function which computes the number of common substructures between two trees T1 and T2. Thus, we can train SVMs with structures drawn directly from the syntactic parse tree of the sentence.</Paragraph>
      <Paragraph position="1"> The kernel that we employed in our experiments is based on the SCF structure devised in (Moschitti, 2004). We slightly modified SCF by adding the headwords of the arguments, useful for representing the selectional preferences.</Paragraph>
      <Paragraph position="2"> For frame detection on FrameNet, we trained our classifier on 46,734 training instances and tested on 11,650 testing instances, obtaining an accuracy of 91.11%. For ILC detection the results are depicted in Table 2. The first six columns report the F1 measure of some verb  class classifiers whereas the last column shows the global multiclassifier accuracy.</Paragraph>
      <Paragraph position="3"> We note that ILC detection is performed better than frame detection on both FrameNet and PropBank. Also, the results obtained on ILC on PropBank are similar with the ones obtained on ILC on FrameNet. This suggests that the training corpus does not have a major influence. Also, the SCF-based tree kernel seems to be robust in what concerns the quality of the parse trees. The performance decay is very small on FrameNet that uses automatic parse trees with respect to PropBank that contains gold parse trees. These properties suggest that ILC are very suitable for free text.</Paragraph>
      <Paragraph position="4"> Table 2 . F1 and accuracy of the argument classifiers and the overall multiclassifier for Intersective Levin class</Paragraph>
    </Section>
    <Section position="3" start_page="83" end_page="83" type="sub_section">
      <SectionTitle>
5.3 Automatic semantic role labeling on
FrameNet
</SectionTitle>
      <Paragraph position="0"> In the experiments involving semantic role labelling, we used a SVM with a polynomial kernel. We adopted the standard features developed for semantic role detection by Gildea and Jurafsky (see Section 2). Also, we considered some of the features designed by (Pradhan et al., 2004): First and Last Word/POS in Constituent, Subcategorization, Head Word of Prepositional Phrases and the Syntactic Frame feature from (Xue and Palmer, 2004). For the rest of the paper we will refer to these features as being literature features (LF). The results obtained when using the literature features alone or in conjunction with the gold frame feature, gold ILC, automatically detected frame feature and automatically detected ILC are depicted in  measure of some role classifiers whereas the last column shows the global multiclassifier accuracy. The first row contains the number of training and testing instances and each of the other rows contains the performance obtained for different feature combinations. The results are reported for the labeling task as the argumentboundary detection task is not affected by the frame-like features (G&amp;J).</Paragraph>
      <Paragraph position="1"> We note that automatic frame results are very similar to automatic ILC results suggesting that ILC feature is a very good candidate for replacing the frame feature. Also, both automatic features are very effective, decreasing the error rate of 20%.</Paragraph>
    </Section>
    <Section position="4" start_page="83" end_page="83" type="sub_section">
      <SectionTitle>
5.4 Semantic role learning curve when us-
</SectionTitle>
      <Paragraph position="0"> ing Intersective Levin classes The next set of experiments show the impact of the ILC feature on semantic role labelling when few training data is available (Figure 3). As can be noted, the automatic ILC features (i.e. derived with classifers trained on FrameNet or PB) produce accuracy almost as good as the gold ILC one. Another observation is that the SRL classifiers are not saturated and more training examples would improve their accuracy.</Paragraph>
    </Section>
    <Section position="5" start_page="83" end_page="83" type="sub_section">
      <SectionTitle>
5.5 Annotating PropBank with FrameNet
</SectionTitle>
      <Paragraph position="0"> semantic roles To show that our approach can be suitable for semantic role free-text annotation, we have automatically classified PropBank sentences with the FrameNet semantic-role classifiers. In order to measure the quality of the annotation, we randomly selected 100 sentences and manually verified them. We measured the performance obtained with and without the automatic ILC feature. The sentences contained 189 arguments from which 35 were incorrect when ILC was used compared to 72 incorrect in the absence of this feature. This corresponds to an accuracy of 81% with Intersective Levin class versus 62% without it.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>