<?xml version="1.0" standalone="yes"?>
<Paper uid="P00-1065">
  <Title>Automatic Labeling of Semantic Roles</Title>
  <Section position="5" start_page="0" end_page="0" type="metho">
    <SectionTitle>
4 Methodology
</SectionTitle>
    <Paragraph position="0"> We divide the task of labeling frame elements into two subtasks: that of identifying the boundaries of the frame elements in the sentences, and that of labeling each frame element, given its boundaries, with the correct role. We #0Crst give results for a system which  ment&amp;quot; from the #5Cconversation&amp;quot; frame labels roles using human-annotated boundaries, returning to the question of automatically identifying the boundaries in Section 5.3.</Paragraph>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
4.1 Features Used in Assigning
Semantic Roles
</SectionTitle>
      <Paragraph position="0"> Thesystem is a statistical one, based on training a classi#0Cer on a labeled training set, and testing on an unlabeled test set. The system is trained by #0Crst using the Collins parser #28Collins, 1997#29 to parse the 36,995 training sentences, matching annotated frame elements to parse constituents, and extracting various features from the string of words and the parse tree. During testing, the parser is run on the test sentences and the same features extracted. Probabilities for each possible semantic role r are then computed from the features. The probability computation will be described in the next section; the features include: Phrase Type: This feature indicates the syntactic type of the phrase expressing the semantic roles: examples include nounphrase#28NP#29, verb phrase#28VP#29, and clause #28S#29. Phrase types were derived automatically fromparse trees generated by the parser, as shown in Figure 2. The parse constituent spanning each set of words annotated as a frame elementwas found, and the constituent's nonterminal label was taken as the phrase type. As an example of how this feature is useful, in communication frames, the Speaker is likely appear a a noun phrase, Topic as a prepositional phrase or noun phrase, and Mediumas a prepostional phrase, as in: #5CWe talked about the proposal over the phone.&amp;quot; When no parse constituent was found with boundaries matching those of a frame element during testing, the largest constituent beginning at the frame element's left boundary and lying entirely within the element was used to calculate the features.</Paragraph>
      <Paragraph position="1"> Grammatical Function: This feature attempts to indicate a constituent's syntactic relation to the rest of the sentence,  Parse constituents corresponding to frame elements are highlighted. for example as a subject or object of a verb. As with phrase type, this feature was read from parse trees returned by the parser. After experimentation with various versions of this feature, we restricted it to apply only to NPs, as it was found to have little e#0Bect on other phrase types. Each NP's nearest S or VP ancestor was found in the parse tree; NPs with an S ancestor were given the grammatical function subject and those with a VP ancestor were labeled object. In general, agenthood is closely correlated with subjecthood. For example, in the sentence #5CHe drove the car over the cli#0B&amp;quot;, the #0Crst NP is more likely to #0Cll the Agent role than the second or third.</Paragraph>
      <Paragraph position="2"> Position: This feature simply indicates whether the constituent to be labeled occurs before or after the predicate de#0Cning the semantic frame. We expected this feature to be highly correlated with grammatical function, since subjects will generally appear before a verb, and objects after. Moreover, this feature mayovercome the shortcomings of reading grammatical function from a constituent's ancestors in the parse tree, as well as errors in the parser output.</Paragraph>
      <Paragraph position="3"> Voice: The distinction between active and passive verbs plays an important role in the connection between semantic role and grammatical function, since direct objects of activeverbs correspond to subjects of passive verbs. From the parser output, verbs were classi#0Ced as active or passive by building a set of 10 passiveidentifying patterns. Each of the patterns requires both a passive auxiliary #28some form of #5Cto be&amp;quot; or #5Cto get&amp;quot;#29 and a past participle.</Paragraph>
      <Paragraph position="4"> Head Word: As previously noted, we expected lexical dependencies to be extremely important in labeling semantic roles, as indicated by their importance in related tasks such as parsing. Since the parser used assigns each constituent a head word as an integral part of the parsing model, we were able to read the head words of the constituents from the parser output. For example, in a communication frame, noun phrases headed by #5CBill&amp;quot;, #5Cbrother&amp;quot;, or #5Che&amp;quot; are more likely to be the Speaker, while those headed by #5Cproposal&amp;quot;, #5Cstory&amp;quot;, or #5Cquestion&amp;quot; are more likely to be the Topic.</Paragraph>
      <Paragraph position="5"> For our experiments, we divided the FrameNet corpus as follows: one-tenth of the annotated sentences for each target word were reserved as a test set, and another one-tenth were set aside as a tuning set for developing our system. A few target words with fewer than ten examples were removed fromthe corpus. In our corpus, the average number of sentences per target word is only 34, and the number of sentences per frame is 732  |both relatively small amounts of data on whichto train frame element classi#0Cers.</Paragraph>
      <Paragraph position="6"> Although we expect our features to interact in various ways, the data are too sparse to calculate probabilities directly on the full set of features. For this reason, we built our classi#0Cer bycombining probabilities from distributions conditioned on a variety of combinations of features.</Paragraph>
      <Paragraph position="7"> An importantcaveat in using the FrameNet database is that sentences are not chosen for annotation at random, and therefore are not necessarily statistically representative of the corpus as a whole. Rather, examples are chosen to illustrate typical usage patterns for each word. We intend to remedy this in future versions of this work by bootstrapping our statistics using unannotated text.</Paragraph>
      <Paragraph position="8"> Table 2 shows the probability distributions used in the #0Cnal version of the system. Coverage indicates the percentage of the test data for which the conditioning event had been seen in training data. Accuracy is the proportion of covered test data for which the correct role is predicted, and Performance, simply the product of coverage and accuracy, is the overall percentage of test data for which the correct role is predicted. Accuracy is somewhat similar to the familiar metric of precision in that it is calculated over cases for which a decision is made, and performance is similar to recall in that it is calculated over all true frame elements. However, unlike a traditional precision#2Frecall trade-o#0B, these results have no threshold to adjust, and the task is a multi-way classi#0Ccation rather than a binary decision. The distributions calculated were simply the empirical distributions from the training data. That is, occurrences of each role and each set of conditioning events were counted in a table, and probabilities calculated by dividing the counts for each role by the total number of observations for each conditioning event. For example, the distribution</Paragraph>
      <Paragraph position="10"> Some sample probabilities calculated from the training are shown in Table 3.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>