<?xml version="1.0" standalone="yes"?>
<Paper uid="C00-2118">
  <Title>Automatic Lexical Acquisition Based on Statistical Distributions*</Title>
  <Section position="4" start_page="815" end_page="815" type="metho">
    <SectionTitle>
2 The Argument Structures
</SectionTitle>
    <Paragraph position="0"> Our approach rests on tile hypothesis that, even in cases where verb classes cannot be distinguished by subcategorizations, the frequency distributions of syntactic indicators can hold clues to the underlying thematic role differences. We start here then with a description of the subca.tegorizations and thematic role assignments for each of l.he three verb classes under investigation.</Paragraph>
    <Paragraph position="1"> As optionally intransitive verbs, each of  the three classes participates in the transitive/intransitive Mternation: Uuergative (la) The horse raced past the barn.</Paragraph>
    <Paragraph position="2"> (1 b) The jockey raced the horse past tile barn. Unaccnsative (2a) The butter melted in the pan.</Paragraph>
    <Paragraph position="3"> (2b) The cook melted the butter in the pan. Object-drop (3a) The boy washed the hall.</Paragraph>
    <Paragraph position="4"> (3b) The boy washed.</Paragraph>
    <Paragraph position="5">  Unergatives are intransitive action verbs, as in (1), whose transitive form can be the causative counterpart of the intransitive form. In the causative use, the semantic argument that appears as the subject of the intransitive, as in (la), surfaces as the object of the transitive, as in (lb) (Ilale and Keyser, 1993). Unaccusatives are intransitive change of state verbs, as in (2a); the transitive counterpart for these verbs exhibits the causative alternation, as in (2b). Object-drop verbs, as in (3), have a non-causative transitive/intransitive alternation, in which the object is simply optional.  Each class is distinguished by the content of tile thematic roles assigned by tile verb. For object-drop verbs, tile subject is all Agent and the optional object is a Theme, yielding tile thematic assignments (Agent, Tlmme) and (Agent) for the transitive and intransitive alternants respectively. Unergatives and uuaccusatives differ \['1&amp;quot;o111 object-drop verbs in participating in the causative alternation, and also differ from each other in their core thematic argument. In an intransitive unergalive, the subject is an Agent, and in an intransitive unaccusative, the subject is a Theme. In the causative transitive form of each, this core semantic argument is expressed as the direct object, with the addition of a Causal Agent (the causer of the action) as subject in bol;h cases. The thematic roles assigned, and their mapping to syntactic position, are summarized in Ta.ble 1.</Paragraph>
  </Section>
  <Section position="5" start_page="815" end_page="816" type="metho">
    <SectionTitle>
3 The features for Classification
</SectionTitle>
    <Paragraph position="0"> The key to any automatic classification task is to determine a set of' useful fea.tures for discriminating the itenls to be classitied. In what follows, we refer to the cohnnns of Table 1 to explain \]tow we expect the thematic distinctions to yield distributional features whose frequencies discriminate among the classes ~t hand.</Paragraph>
    <Paragraph position="1"> Considering column one of Table 1, only unergative and unaccusa.tive verbs assign ~ Causal Agent to the subject of the transitive. We hy1)othesize that the causative construction is linguistically more complex than the simple argument optionality of object-drop verbs (Stevenson and Merlo, 1.997). We expect then that object-drop verbs will be more fi:equent in the transitive than the other two classes. Furthernmre, the object of an unergative verb receives the Agent role (see the second column of Table 1.), a linguistically marked transitive construction (Stevenson and Merlo, 1997). We therefore expect unergatives to be quite rare in the transitive, leading to a three-way distinction in transitive usage among the three classes.</Paragraph>
    <Paragraph position="2"> Second, due to the causative alternation of  TrmlsitiviLy Unaccusativcs and unergativcs have ~, causative transitive, hence lower transitive use. Furl;hc'rlnorc, unerga.tivcs ha.re a.n agent.ire object, hence very low transitive use. Pa.ssivc Voice Passive implies transitive use, hence correlated with transitive feature. VBN Tag Passive implies past pa.rt;iciple use (VBN), hence correlated with transitive (and passive). Causativity ()l)ject-drop verbs do not have a. causal agent, hence low &amp;quot;ca.usative&amp;quot; use. Unergatives are rare in the transitive, hence low cmlsative use.</Paragraph>
    <Paragraph position="3"> Animacy Unaccusatives have a Theme subject in the intransitive, hence lower use of animal, e subjects. unergatives and nnaccusatives, the l, hematic role of the subjec~ of the intransitiw,~ is identical to that of the objecl of the transitiw;, as shown in columns two and three of Table 1. C, iven the identity of thematic role mal)ped to subject and object positions, we expect to observe the sa.me noun occurring at times a.s subject of the verb, and at other times as object of the verb. In contrast, for object-tirol) verbs, Cite thenm.tic role o\[' the sul)ject o17 the intransitive is identical to l;ha{, of the sul)ject of the transitive, not the object of the transitive. Thus, we expect that it will be less common for the same noun to occur in subject and object position of the same object-drop verb. We hypothesize that this pattern of thematic role assignments will be retlected in difl'erential amount of u~'~age across the classes of the same nouns as subjects and ol)jects for a given verb. Furthermore, since the causative is a transitive use, a.nd the 1,ra.nsitive use of u nerga.gives is oxpocl;ed to be rare., this overlap o(' subjects and ob.iects should primarily distinguish unaccusatives (predicted to have high overlap of subjects and objects) from the other two classes.</Paragraph>
    <Paragraph position="4"> Finally, considering columns one and three of Tal)le 1, we note that unergative and objecl;-drop verbs assign all agentive role to their subject in both the transitive and intra.nsitive, while unaccusatives assign an agentive role to their subject only in the tr~msil, ive. Under the assutnpLion that the intransitive use of' unaccusatives is not rare, 1 we then expect thai, unaccusatives will occur less often overall with an agentive subject than the other two verb classes. On the flu:ther assumption that Agents tend to be animate entities more so than Themes, we expect that unaccusatives will occur less freqnently with an animate subject compared to unergative and object-drop verbs.</Paragraph>
    <Paragraph position="5"> Note the importance of our use of frequency distributions: the claim is not that only Agents can ~This assumpl, ion is based on the linguistic conlplexity of the causative, and borne out in our corpus analysis.</Paragraph>
    <Paragraph position="6"> be animate, but rather that nouns that receive an Agent role will more often be animate than nouns that receive a Theme ,'ole.</Paragraph>
    <Paragraph position="7"> The above interactions between thematic roles and the syntactic expressions of arguments thus lead to three features whose distrit)utional properties appear promising for distinguishing the verb classes: transitivity, causativity, and animaey of subject. We also investigate two additionM syntactic l'ea.l, ures, the passive voice and tile past pa.rticiple POS tag (VI3N). These features are related to the transitive/intransitive Mternal;ion, since a passive use implies a transitive use of the verb, and the nse of passive in turn implies the use of the past participle. Our hyl)ol;hesis is that these five features will exhibit distributional differences in the observed usages of the verbs, which can be used for classifica.tion. The features and their expected relevance are summarized in '13ble 2.</Paragraph>
  </Section>
  <Section position="6" start_page="816" end_page="817" type="metho">
    <SectionTitle>
4 Da~a Collection and Analysis
</SectionTitle>
    <Paragraph position="0"> We chose a set of 20 verbs from each of three classes. The complete list of verbs is reported in Appendix A. Recall that our goal is to achieve a fine-grained classification of verbs that exhibit the same subcategorization frames; thus, tile verbs were chosen because they do not generally show massive del)artures from the intended verb sense (and usage) in the corpus. 2 In order to simplify tile counting procedure, we included only tile regular (&amp;quot;-ed&amp;quot;) simple past/past participle form of tile verb, assuming that this would approximate the distribution of tile features across all forms of the verb. Finally, as far as we were able given the preceding constraints, we selected verbs that could occur in the transitive and in the passive.</Paragraph>
    <Paragraph position="1"> We counted the occurrences of each verb token in a transitive or intransitive use (3'RANS), ill a 2~l~hough note that there are only 19 unaccusatives because ripped was excluded fl'om the analysis as it occurred mostly in a very different use (ripped off) in the corpus from the intended diange of state usage.</Paragraph>
    <Paragraph position="2">  passive or active use (PASS), in a past participle or simple past use (VBN), in a causative or non-causative use (tAgS), and with an animate subject or not (ANIM), as described below. The first three counts (TRANS, I'ASS~ VBN) were performed on the LDC's 65-million word tagged ACL/DCI corpus (Brown, and Wall Street Journal 1987-1989).</Paragraph>
    <Paragraph position="3"> The last two counts (CAUS and ANIM) were performed on a 29-million word parsed corpus (\gall Street Journal 1988, provided by Michael Collins (Collins, 1997)). The features were counted as follows: TaANS: The closest noun following a verb was considered a potential object. A verb immediately \[bllowed by a potential object was counted as transitive, otherwise as intransitive.</Paragraph>
    <Paragraph position="4"> pass: A token tagged VBD (the tag for simple past) was counted as active. A token tagged VBN (the tag for past participle) was counted as active if the closest preceding auxiliary was have, and as passive if the closest preceding auxiliary was be.</Paragraph>
    <Paragraph position="5"> VBN: The counts tbr VBN/VBI) were based on the POS label in the tagged corl)us.</Paragraph>
    <Paragraph position="6"> Each of the above counts was normalized over all occurrences of tim &amp;quot;-ed&amp;quot; form of the verb, yielding a single relative fi:equency measure \['or each verb for that feature.</Paragraph>
    <Paragraph position="7"> tags: For each verl) token, the subject and object (it' there was one) were extracted from the parsed corpus, and the proportion of overlap between subject and object nouns across all tokens of a verb was calculated.</Paragraph>
    <Paragraph position="8"> ANIM: To approximate animacy without reference to a resource external to the corpus (such as WordNet), we count pronouns (other than it) in subject position (cf. (Aone and McKee, 1996)).</Paragraph>
    <Paragraph position="9"> The aSSUlnption is that the words I, we, you, .~'tze, he, and theft most often refer to animate entities.</Paragraph>
    <Paragraph position="10"> We automatically extracted all subject/verb tuples, and computed the ratio of occurrences of pronoun subjects to all subjects for each verb.</Paragraph>
    <Paragraph position="11"> The aggregate means by class resulting from the counts above are shown in Table 3. The distributions of each feature are indeed roughly as expected according to the description in Section 3.</Paragraph>
    <Paragraph position="12"> Unergatives show a very low relative fi'equency of the TRANS feature, followed by unaccusatives, then object-drop verbs. Unaccusative verbs show a high frequency of the CAUS feature and a low frequency of the ANIM feature compared to the other classes. Although expected to be a redundant indicator of transitivity, pass and VBN do  not distinguish t)etween unaccusative and object-drop verbs, indicating that their distributions are sensitive to factors we have not yet investigated, a</Paragraph>
  </Section>
  <Section position="7" start_page="817" end_page="819" type="metho">
    <SectionTitle>
5 Experiments in Classification
</SectionTitle>
    <Paragraph position="0"> The frequency distributions of our features yield a vector for each verb that represents the relative frequency wdues for the verb on eacln dimension: \[verb, TRANS, PASS, VBN~ CAUS, ANIM, class\] Example: \[opened, .69, .09, .21, .16, .36, unaec\] \Y=e use the resulting 59 vectors to train an automatic classifier to determine, given a verb that exhibits transitive~intransitive sttbcategorization frames, which of the three major lexical semantic classes of English optionally intransitive verbs it belongs to. Note that the baseline (chance) per-Ibrmance in this task is 33.9%, since there are 59 vectors and 3 possible classes, with the most coinmen class having 20 verbs.</Paragraph>
    <Paragraph position="1"> We used the C5.0 machine learning system (tnttp://www.rulequest.com), a newer version of C4.5 (Quinlan, 1992), which generates decision trees and corresponding rule sets from a training set of known classifications. We found little to no difference in performance between the trees and rule sets, and report only the rule set results. \Y=e report here on experiments using a single hold-out training and testing methodology. In this approach, we hold out a single verb vector as the test case, and train the system on the remaining 58 cases. We then test the resulting classifier on tile single hold-out case, recording tile assigned class for that verb. This is then repeated for each of the 59 verbs. This technique has the benefit of yielding both an overall accuracy rate (when the results are averaged across all 59 trials), as well as providing tile data necessary tbr determining accuracy for each verb class (because we have the classification of each verb when it is the test case). This allows us to evaluate tile contribution aThese observations have been confirmed by t-test.s between feature values for each pair of classes.</Paragraph>
    <Paragraph position="2">  1. '.I'P VCAn 69.5 85.0 2. P V C An 64:.4: 80.0 3. TV C An 71.2 80.0 4:. 51' P C An 61.0 65.0 5. TPVAn 62.7 70.0 6. 51'PV C 61.0 80.0  of individual feal:ures with respect to their effect on the perfornlance of individual classes.</Paragraph>
    <Paragraph position="3"> We performed experiments on the \['ull sel, of features, as well a.s each subsel, of fea.l,ures wil,h a. single f~ture remow;d, as reported in Table d. Consider l;he first column in the ta.ble. The first line shows that the overall ~ccuracy for all live features is 69.5%, a reduction in tile error ra.te of more than 50% above the baseline. The removal o\[&amp;quot; the PASS lea.lure appears to improve performance (row 3 of Ta.ble 4). However, it should be noted that this increase in performance results h:oln a single additionaJ verb being classified correctly. The rema.ining rows show thal no feal,ure is superflous or hm'mfld as l,he removal of any I'ealure has a. 5 8% negative elfect on l)erR)rmance. Coral)arable.</Paragraph>
    <Paragraph position="4"> accuracies have been demonsl;rated vsing a more thorough cross-validation methodology a.nd using reel;hods that are, in principle, better a,t taking adva.nl,age of correlated lea,lures (Stevenson and Merle, 1999; Stevenson el. al., 1999).</Paragraph>
    <Paragraph position="5"> q'he single hold-out prol,ocol provides new data, fbr analysing the performalme on individual verbs and classes. The class-by-class accuracies a.re shown in the remaining columns of Ta.ble 4. \Y=e can see clearly thal, using all five features, l,he unergatives are classified with much greater accuracy (85%) than l,he UlmCCUsatives and object-drop verbs (63.2% and 60.0% respectively), as shown in the first row. The rema.ining rows show that this l)al,tern generally holds \['or l,he subsel,s of features as well, with tire excel)lion of line d. \Y=hile ful,ure work on our verb classificalion task will need lo focus on deterlnining features thal bel,ter discriminate unaccusative a.nd object-drop verbs, we can ah:eady exclude an explanation of the resull,s based simply on l,he wwbs' or tile classes' frequency. Unergatives have tile lowest average (log) frequency (1.3), but are the best classified, while unaccusatives and object-drops are comparable (a.verage log fi'equency = 2). If we group verbs by frequency, the proportion of errors to lhe total number of verbs remains fairly similar (freq 1:7 errors/23 verbs; fi:eq. 2:6 errors/24 verbs; freq. 3:4 errors/10 verbs). The only verb of frequency 0 is correctly classified, while lhe only one with log frequency 4 is not . In sum, we do not find that more frequent classes or verbs are more accurately classitied.</Paragraph>
    <Paragraph position="6"> lmlmrtantly, the experiments also enable us to see whether the fealures indeed contribute to discriminating the classes in the manner predicted in Seclion 3. The single hold-out results allow us to do 1;his, by comparing the individual class labels assigned using the full sol, of five features (TIIANS, PASS, VBN, CAUS, ANIM) to the class labels assigned using each size four subset of features. This comparison indicates 1;he changes in class labels l,hat we can a.l,tribul,e to l,he added feature in going fi'om a size four subset to the full set of features. (The individual class labels supporling our a.nalysis below a.re a.vailable from the authors.) \Y=e concent;rate on tile three main features: CAUS, ANIM, TRANS. \Y=e filial thai, the behaviour of lhese feal,ures generaJly does conform to our predicl;ions. We expected that TRANS would help make a. three-way distinction among the verb classes. While unergatives are ah:eady accurately classified with-Ollt TRANS, inspection of lhe change in class la.bels reveals that the addition of TRANS tO tire sel; improves performance on unaccusatives by helping to distinguish 1;hem from object-drol)s, llowever, in this case, we also observe a loss in precision of unerga.lives, since some object-drops are now classitied a.s unergatives. Moreover, we expected CAUS and ANIM tO be parl,icularly helpfid in identi\['ying unaccus~l,ives, and this is also borne out in our analysis of individual la.bels. We note that the increased accuracy from CAUS is primarily due to bel,ter disl,inguishing unergatives from unaccusatives, and l,he increased accura.cy from AN1M is primarily due go better distinguishing unaccusatives from objecl,-drops. \Y=e conclude tha.t the feal,ures we have devised are successful in classiting optionally 1,ra.nsil;ive verbs because they ca.plure predicted difl'erences in underlying argument structlrre. 4 4 Matters are more cmnplex with the other two features and we arc still interpreting tile results. Our prediction</Paragraph>
  </Section>
  <Section position="8" start_page="819" end_page="819" type="metho">
    <SectionTitle>
6 Comparison to Experts
</SectionTitle>
    <Paragraph position="0"> In order to evaluate the performance of the algorithm in practice, we need to compare it to the accuracy of classification performed by an expert, which gives a realistic upper bound for the task.</Paragraph>
    <Paragraph position="1"> In (Merle and Stevenson, 2000) we report the resuits of an experiment that measures experts pertbrmance and agreement on a classification task very similar: to the program we have described here. The results summarised in Table 5 illustrate the performance of the progra, m. On the one hand, the algorithm does not perform at expert level, as indicated by the fact that, for all experts, the lowest agreement score is with the program. On the other: hand, the accuracy achieved by the program of 69.5% is only 1.5% less than one of the human experts in comparison to tire gold standard. In fact, if we take the best performance achieved by an expert in this task 86.5%--as the maximum achievable accuracy in classification, our algorithm then reduces the error rate over; chance by approximately 68%, a very respectable result.</Paragraph>
  </Section>
  <Section position="9" start_page="819" end_page="820" type="metho">
    <SectionTitle>
7 Discussion
</SectionTitle>
    <Paragraph position="0"> The work here contributes both to general and technical issues in automatic lexical acquisition.</Paragraph>
    <Paragraph position="1"> Firstly, our results confirm the primary role of argument structure in verb classification. Our experimental focus is particularly clear in this regard because we deM with verbs that are ~Illilliwas that VBN and PASS would behave similarly to TRANS. In fact, PASS is at best unhelpful in classification. VBN does appear to make the expected I.hree-way distinction.</Paragraph>
    <Paragraph position="2"> The change ill class labels shows that the improvement in performance with VBN results from better distinguishing unergatives fi'om object-drops, and object-drops from unaccusatives. The latter is surprising, since analysis of the data found that the VnN feature values are statistically indistinc~ for the object-drop and unaccusative classes as a whole.</Paragraph>
    <Paragraph position="3"> mal pairs&amp;quot; with respect to argument structure. 13y classif~ying verbs that show the same subcategorizations into different classes, we are able to eliminate one of the confounds in classification work created by the fact that subcategorization and argument structure :M'e largely co-variant. We can infer that the accuracy in our classification is due to argument structure information, a.s subcategorization is the same for: all verbs, confirming that the con, tent of thematic roles is crucial for classification. Secondly, our results further support the assumption that thematic differences such as these are apparent not only in differences in sub-categorization frames, but also in differences in (;heir frequencies. We thus join the many recent results that all seem to converge in SUl)porting the view that the relation between lexical syntax and semantics can be usefully exploited (Aone and McKee, 1996; l)orr, 1997; Dorr and Jones, 1996; Lapata and Brew, 1999; Schulte im Walde, 1998; Siegel, 1998), especially in a statistical franmwork.</Paragraph>
    <Paragraph position="4"> Finally, we observe that this information is detectable in a corpus and can be learned automatically. Thus we view corpora, especially if annotated wil;h currently available tools, a.s useful repositories of implicit grammars.</Paragraph>
    <Paragraph position="5"> Technically, our N)proach extends existing corpus-based learning techniques 1;o a more complex lea.ruing problem, in severaJ dimensions. Our statistical apl)roach , which does not require explicit negative examples, extends ai)l)roaehes that encode l~evin's alternations directly, as symbolic properties of a verb (Dorr et al., 1995; l)orr and Jones, 1996; l)orr, 1997). We also extend work using surface indicators to approximate underlying properties. (Oishi and Matsumoto, 1997) use case marking particles to approximate graimnatical functions, such as subject and object. We improve on this approach by learning argument structure properties, which, unlike grammatical functions, are not marked lnorphologically. Others have tackled the problem of lexical semantic classification, as we have, but using only snbeategorization frequencies as input data (Lapata and Brew, 1.999; Sehulte im Walde, 1998). By contrast, we explicitly address the definition of features that can tap directly into thematic role differences that are not reflected in &amp;quot;subcategorization distinctions. Finally, when learning of thematic role assignment has been the explicit goal, the text has been semantically annotated (Webster and Marcus, 1989), or external semantic re- null sources ha.re I)een consulted (Ache and McI(ee, 19!)6). We extend these results by showing that them;~tic informa,tion can 1)e inducexl from corpus COtllltS.</Paragraph>
    <Paragraph position="6"> The exl)erimental results show that our method is l)owerful, and suited to Cite classitica.l;i(m of lexica.1 items. However, we have not yet addressed the problem of verbs that can h;~ve multiple classifications. We think tha.t many eases of am1)iguous classification of verb types can 1)e addressed with the notion of intersective sets introduced by (Da.ng et al., 71998). This is an imt)ortant concept tha,t l)rOl)OSes tha,t &amp;quot;i'egula, r&amp;quot; a.mbiguity in classifica.tion -i.e., sets of v(;rbs that ha.ve the same multi-way classitications a~ccording to (l,evin, 1993) can be captured with a. linergrained notion of lexical semantic classes. I~xtending our work to exploit this idea. requires only to define the classes a.pl)ropriately; the basic a.1)t)roac\]l will remain the same. When we turn to consider ambiguity, we must a.lso address the l)roblem l;ha.t individual insta.nces of verl)s may come from diffel:ent classes. In future research we t)lan to extend our method to the ('\]a.ssificagion of a.mbiguous tokens, by experimenting with a. functics that combines severaJ sources of information: a bias tbr the verb type (using the cross-corpus sta.l;istics we collect), as well as \[~a.tures o\[&amp;quot; the usage of the insta.nce being classiiiod (cf. (l,apa.ta a,n&lt;l I~rew, t999; Siegel, 199,q)).</Paragraph>
  </Section>
  <Section position="10" start_page="820" end_page="820" type="metho">
    <SectionTitle>
Appendix A
</SectionTitle>
    <Paragraph position="0"> Unergatives: floated, galloped, glided, hiked, hopped, hurried, jogged , jumped, leaped, marched, paraded, raced, rushcd, seootcd, scurricd, skipped, tiptoed, t~vttcd, va,dtcd, wandered. Unaccusativcs: boiled, changcd, cleared, collapsed, cooled, cracked, dissolved, divided, exploded, flooded, .folded, fractured, hardcned, melted, opcncd, simmcred, solidified, stabilized, widened. Object-drops: bof rowed, eallcd, earvcd, clca~cd, danced, inheritcd, kiekcd, knittcd, or aniscd, packcd, paintcd, playcd, reaped, rcnted, skclehe.d, studied, swallowed, typed, washcd, ycllcd.</Paragraph>
  </Section>
class="xml-element"></Paper>