<?xml version="1.0" standalone="yes"?>
<Paper uid="W98-1116">
  <Title>What grammars tell us about corpora: the case of reduced relative clauses</Title>
  <Section position="5" start_page="134" end_page="135" type="metho">
    <SectionTitle>
2 Reduced Relative Clauses
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="134" end_page="135" type="sub_section">
      <SectionTitle>
2.1 Linguistic Properties
</SectionTitle>
      <Paragraph position="0"> The following classic &amp;quot;'garden-path&amp;quot; example demonstrates tlm sew:re processing difficulty that can be associated with the main verb/reduced relatiw: (MV/RR) ambiguity (Bever, 1970): (1) The horse raced past the barn fell.</Paragraph>
      <Paragraph position="1"> Problems arise here because the vcrb raced can be interpreted as either a past tense main verb, or as a past participle within a reduced relative clause (i.e., the hor.s'tC/ \[that was\] raced past the barn). Because fell is the main verb of (1), the reduced relative interpretation of raced is required for a coherent analysis of the complete sentence. But the inain verb interpretation of raced is so strongly preferred that the human language processor breaks down at the verb fell, unable to integrate it with the interpretation that has been developed to that point.</Paragraph>
      <Paragraph position="2"> This construction is representative of the problem we want to address. It. is very frequent (MacDonald et al., 1994). hence it constitutes a problem that is relevant for any application. It is both lexically and structurally ambiguous, so it constitutes a hard problem. It is well-studied: there are plentiful data on lmman processing and their relation to fi'equency of the stimuli (MacDonald, 1994; Trueswell. 1996: Trueswell</Paragraph>
    </Section>
  </Section>
  <Section position="6" start_page="135" end_page="138" type="metho">
    <SectionTitle>
VERB TYPE EXAMPLE .JUDGMENT
</SectionTitle>
    <Paragraph position="0"> unergative The horse raced past the barn fell hard unaccusative The butter melted m the pan was rancid easy object-drop The player kicked i:a the soccer game was angry easy  et al., 1994).</Paragraph>
    <Paragraph position="1"> Over the last several years, it has become clear that not all reduced relatives are as difficult as sentence (1) above, and that the difficulty in processing reduced relatives is directly linked to the lexical items in the sentence. In particular the difficulty appears to be related to the type of verb which is involved in the ambiguity. For the ambiguity to arise, the w.'rb involved--raced in this case--must be optionally transitive. English has three types of optionally transitive verbs, which differ both in their lexical semantics and in their syntactic properties.</Paragraph>
    <Paragraph position="2"> Sentence (1) uses a manner of motion verb, raced. In English, these verbs form a subclass of mmrgative verbs (Levin and Rappaport Hovav, 1995), intransitive action verbs that may appear in a transitive form: (2a) The horse raced past the barn.</Paragraph>
    <Paragraph position="3"> (2b) The rider raced the horse past the barn.</Paragraph>
    <Paragraph position="4"> The transitive form of an unergative (2b) is the causative counterpart of the intransitive form (2a), in which the subject of the intransitive becomes the object of the transitive (Hale and Keyser: 1993; Levin and Rappaport Hovav, 1995). Sentences (3a) and (3b) use an unacmlsative verb. melt: (3a) The butter melted in the pan.</Paragraph>
    <Paragraph position="5"> (3b) The cook melted the butter in the pan.</Paragraph>
    <Paragraph position="6"> Unaccusatives are intransitive change of state verbs which also have a causative transitive form. They differ from unergatives because their alternating theta role is a theme (butter), while for unergatives it is an agent (horse). Finally, sentences (4a) and (4b) use. an object-drop verb. kicked; these verbs have a non- null causative transitive/intransitive alternation, in which the object NP is optional: (4a) The player kicked the referee.</Paragraph>
    <Paragraph position="7"> (4b) The player kicked.</Paragraph>
    <Section position="1" start_page="135" end_page="135" type="sub_section">
      <SectionTitle>
2.2 Processing Difficulty
</SectionTitle>
      <Paragraph position="0"> (Stevenson and Merlo. 1997) asked naive informants for acceptability judgments on sentences with reduced relatives (RRs) containing these verbs. They found that unergative verbs, such as raced or j,m,ped, unitbrmly led to a severe garden path in tim R R construction. while unaccusativc vm'l)s were ()verwhehningly judged completely fine in the R R.. with a few responses of them I,,~ing; slightly degraded They did not ask tbr .iu,lgments on (fl~ject-drop verbs; native speakers&amp;quot; intuitions are that they are readily interpretabh', in ;t RR. Supl)ort for this view comes fi'om CXl)Criments which included object-drop verbs, that showed that R.R.s are relatively easy to ,rodin'stand given a context that is not strongly I)iased toward a main verb reading (MacDonakl. 1994). Tlms. the dig ficulty of the RR intcrpret}ttion l);ttterns along verb class lines, with ,m,.rgatives difficult, and unaccusatives and obj,,,:t-drol~ vm'bs relatively easy. V~re summarise these results in Table 1.</Paragraph>
    </Section>
    <Section position="2" start_page="135" end_page="137" type="sub_section">
      <SectionTitle>
2.3 Statistical Properties
</SectionTitle>
      <Paragraph position="0"> We measured the prolmbility distrilmtions of several linguistic t(.';ttures (transitivity: tense, voice) over a sample of optionally transitive verbs fi'om the three lcxical semantic classes described above. We proceeded by hYlmthesizing and testing several probability flmctions over the sample, and proposing an ev,mt ,:lassification that best fits the native sp,,aker judgments described above.</Paragraph>
      <Paragraph position="1"> In our view. a grammar is a wav ,ff classifying elements in a language. Our sample of language is a text, and our grammar is the space of elementary events we define on the text. So our grammar is the space, of events over which we calculate tim probability distributions. The emphasis on lexicalised grammars, both in linguistics, sentence processing and statistical NLP, points towards statistics ,:Omlmted at the level  of lexical items or their subfeatures.</Paragraph>
      <Paragraph position="2"> A probability space is a triple ~, .T', P, where f2 is the sample space, .T&amp;quot; is the event space and P is a function P : F ~ \[0, 1\]. In the discussion below, we assume 5 different probability spaces, in which the event space is defined by sublexical properties of verbs.</Paragraph>
      <Paragraph position="3"> First, we counted the occurrences of the verbs as a simple past main verb (MV) and the occurrences of the verbs as a reduced relative (RR). Second, we counted the occurrences of the verbs in a transitive (TRANS) or intransitive (INTR) form. Third, we counted the occurrences of the verbs in an active (ACT) or passive (PASS) form. Then, we counted the occurrences of the verbs as a simple past main verb (MV) and the occurrences of the verbs as a past participle (PRT). These features were chosen because they nfinimally distinguish main clause from reduce relative forms. Finally, we counted how often the past participle form was used adjectivally.</Paragraph>
      <Paragraph position="4"> This last count was chosen because only certain lexical semantic classes of verbs (excluding mmrgative verbs) can occur as adjectives (Levin and Rappaport 1986).</Paragraph>
      <Paragraph position="6"> that the probabilities of the events are indicated by their relative frequency.</Paragraph>
      <Paragraph position="7"> We test the following hypothesis: H0: differences in processing preferences correspond to differences in the distributions of the measured variables. null  We chose a set of 10 verbs from each class, based primarily on the classification of verbs in (Levin. 1993): the unergatives are manner of motion verbs (jumped, rushed, marched, leaped, floated, raced, hurried, wandered, vaulted, pafaded), the unaccusatives are verbs of change of state (opened. ezploded, flooded, dissolved, cracked, hardened, boiled, melted, fractured, so-Iidified), and the object-drop verbs are unspecified object alternation verbs (played. painted, kicked, carved, reaped, washed, danced, yelled, typed, knitted). Each w;rb presented the same form in the simple past and in the past participle, as in the MV/RR ambiguity. All verbs can occur in the transitiw~, and in the passive. The verbs in the three sets were matched pairwise in frequency, and their logarithmic fi'cquency varies between 2 and 4 ilmlusive.</Paragraph>
      <Paragraph position="8"> In performing this kind of corpus analysis, one has to take into accomlt the fact that current corpus annotations do not distinguish verb senses. The verbs in the materials were chosen because they did not show massive departures from the intended verb sense: tbr example, in a different study run w;u~ eliminated because it occurs ulost often in phrases such ;us run a meeting, where it is not a manner of motion use. However, in these comlts, we did not distinguish a core sense of the verl) fi'om an cxtendcd use of the verb. So. for instance, the sentence Consumer spending jumped 1.7 ~ in February after a sharp drop the month before (from Wall Street Journal 1987) is counted as an occurrence of the manner-of-motion verb j'amp in its intransitive form. This is an assmnption that is likely to introduce more variance than if we had only counted core senses of these verbs, but it is an unavoidable limitation at the current state of annotation of corpora.</Paragraph>
      <Paragraph position="9"> Counts were performed on the tagged version of the Brown Corpus and on the portion of the Wall Street Journal distributed by the ACL/DCI (years 1987, 1988, 1989), a combined corpus in excess of 65 million words. Five pairs of counts were collected, for which the raw aggregated results are shown in Table 2. First, each verb was counted in its main w.,rb (i.e., simple past) and past participle uses. based on  the part of speech tag of the verb in the corpora. Second, active and passive uses of the verbs were counted: cases in which usage could not 1)e determined by a simple pattern search w,;re classified by hand. The third count also r,:quired manual intervention: verbs were initially classified as transitive or intransitive ac,:ording to a set of regular search patterns, then individual inspection of verbs was carried out to correct item-specific errors. In the fourth count, uses of the verb form as inain verb or ms reduced relative were collected. Reduced relatives were ,:o,mtcd by hand after extracting fi'om the corpus all occurrences of the past participle preceded by a noun. In the fifth count, uses of the verbs as prenominal adjectives were counted.</Paragraph>
      <Paragraph position="10"> None of the verb forms are explicitly marked as adjectives in these corpora. To deternfine the ,xmnts of adjectival uses, we simply divided the verb occurrences labelled with the past participle part of speech tag into prenominal and other uses. The only unexpected result we found was the occurrence of unergative adjectival forms.</Paragraph>
      <Paragraph position="11"> On inspection all these forms occurred with two verbs: hurried occurred 20 times, and rushed once. These were not the causative use of the verb. So these verbs were removed from the analysis of variance reported below. The unac,:usatives and object-drops that were matched in frequency to hurried and rushed were also removed (unaccusatives: boiled, fractured; objectdrop: danced, typed).</Paragraph>
    </Section>
    <Section position="3" start_page="137" end_page="138" type="sub_section">
      <SectionTitle>
2.4 Results and Discussion
</SectionTitle>
      <Paragraph position="0"> The raw aggregated data in Table 2 show that properties related to the main verb (MV) usage--intransitivity, active voice, non-adjectival use and simple past use. as well as the MV construction itself--were more frequent for unergatives than for unaccusatives, and more frequent for unaccusatives than tbr object-drop verbs. The mnnerical trend is in accord with the simplest explanation on the use of frequency by humans: more fl'equently occurring structures are preferrcd over less fl'equent alternatives. However, not all numerical differences are significant, as indicated in Table 3.</Paragraph>
      <Paragraph position="1"> The data in Table 2 were entered in 10 different analyses of variance on the proportion of cases that indicate a use of the verb as a main verb and its related lexical and sublexical properties -- simple past. active, intransitive and non-adjectival use. Results of the ANOVAs are shown in Table 3. The ANOVAS were run to determine if verbs that belong to a class have a significantly different distribution than verbs that belong to one of the other two classes. We chose to perform analysis of variance because this test compares variance within a group to variance between groups, thus it is not distorted by the fact that there is great w~riation from lexical item to lexical item within each group.</Paragraph>
      <Paragraph position="2"> A simplified summary of the res,flts and the corresponding human intuitive dat}t is given in  All the data sets show the same pattern. For the lexical features-- simple past. active, intransitive and non-adjectival use -- the differences between the unergative and unaccusative distributions for each property are lfighly significant (p &lt; 0.05), but the differences between  ~ws' intuitions compared to the results of significance test on pairwise comparisons of corpus data the unaccusative and object-drop distributions are not (p &gt; 0.05). This could explain why the unergatives are significantly more difficult in the RR, while the other classes of verbs are not perceived as different.</Paragraph>
      <Paragraph position="3"> Interestingly, a more direct count of the construction itself (the MV/RR probability space) gives different results. Numerically, the counts of RR for unaccusatives arc very small, but native speakers do not find RR with unac,:usative verbs particularly difficult. Stati.sti,:ally, mmrgatives are not significantly differ,mr from unaccusatives (p = 0.059), but native speakers find RR with unaccusative verbs considerably easier than with unergatives.</Paragraph>
      <Paragraph position="4"> The picture that emerges from these findings is coherent and in accordance with current dew~lopments in statistical parsing and grammati,:al theory in two important respects. First, the discrepancy between the frequencies of each of the lexical features and the frequencies of the m:tual construction suggests that the frequency of a construction is a composition fimction of (at least some of) its lexical features, even if such t};atures are not-independent. Models that can handle non-independent lexical features have given very good results both for part-of-speech and structural disambiguation (Ratnaparkhi, 1996; Ratnaparkhi, 1997; Ratnaparkhi, 1998).</Paragraph>
      <Paragraph position="5"> Second, we observe that the lexical and sub-lexical features we counted are not sufficient to identify all the relevant linguistic classes: statistical tests fail to differentiate between unaccusatives and object-drop verbs. In order to distinguish between these two classes of verbs one needs to look at some of the surrounding context. This result is expected. Performance measures of statistical parsers show that statistics based on one word give poor results, but that statistics on bigrams have much better performance (Charniak, 1997).</Paragraph>
    </Section>
  </Section>
  <Section position="7" start_page="138" end_page="140" type="metho">
    <SectionTitle>
3 General Discussion
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="138" end_page="138" type="sub_section">
      <SectionTitle>
3.1 Relationship between Different
Kinds of Methods
</SectionTitle>
      <Paragraph position="0"> Our results cast some light on an important methodological question: can frequencies in annotated corpora be considered a good approximation of speakers' preferences? Recent results in the literature have argued that they cannot, showing large discrepancies between data collection methods (Merle, 1994), ,:omprehension and production (Gibson et al.. 1996), and on-line preferences and corpora counts (Brysbaert et al., 1998). Several explanations have been proposed, mostly dismissive of some particular method to collect data: tbr example: frequency-based preferences are not used by hmnans; the wi'ong frequencies had been ,:omltcd: experimental results are not representative of natural linguistic behaviour: or corpora are not representative of natural linguistic behaviom'. The findings in this study show a way of reconciling results obtained by different data collection methods: if we count at the level of lexical and sublexical features, we find that differences in native speakers' preferences do correspond to significant differences in distributions. Similar conclusions are being reached in (Roland and Jurafsky, 1998), who compare different corpora.</Paragraph>
    </Section>
    <Section position="2" start_page="138" end_page="139" type="sub_section">
      <SectionTitle>
3.2 Classification Properties of Lexical
Features and Consequences
</SectionTitle>
      <Paragraph position="0"> Looking at the frequencies of the Iexical features in Table 2, we can observe that P12T, PASS and TRANS have counts that can be used to directly predict the difficulty of the 1212. construction. This observation can be used beneficially in a task different fl'om parsing, for instance in a generation system. Some current meth- null ods have a generate and filter approach (Knight and Hatzivassiloglou, 1995): all constructic,ns are generated and then filtered based on a sl~atistical model. If the trigram model has a good fit with text, our experiments indicates that it would eliminate many RRs for unaccusatives that would be considered acceptable by speakers. If instead the filtering is based on, for example, the frequency of the past participle use, the system would correctly allow unaccusative RRs, but filter out unergative RRs.</Paragraph>
      <Paragraph position="1"> Moreover, we notice that all the lexical features reproduce the well-known relation between markedness within a language and typology of languages: what is an existing but infrequent construction in a few languages is absent in many languages. In this instance, the transitive use of manner of motion verbs -- The rider raced the horse -- is a marked construction in English, in the sense that while it is grammatical. this use is only restricted to a subset of manner of motion verbs. This construction which is marked in English is ungrammatical in R.olnance: languages such as Italian or French do not have a grammatical direct translation for the sentence above. This is called in the social sciences a zero-rare distribution, where a feature that is generally already rare is however never present in one subclass of the eases.</Paragraph>
      <Paragraph position="2"> Interestingly, the lexical feature ADJ presents a distribution that reflects this cross-linguistic t;act internally to English: unergative verbs never occur prenominally, even those that can occur transitively, passively and in reduced relative clauses.</Paragraph>
      <Paragraph position="3"> This is a particularly useful distributional cue tbr verb classification. On observing the complete absence of prenominal adjectives derived from transitive verbs one can classify the verb as unergative. Or, the cue provided can be used in a translation task: one of the typical argument structure divergences between English and Romance languages can be inferred by looking at distributional data. Thus, by observing the absence of prenominal adjectives in English the translation system can avoid proposing the RR alternative in the target language, where it would be ungrammatical.</Paragraph>
    </Section>
    <Section position="3" start_page="139" end_page="140" type="sub_section">
      <SectionTitle>
3.3 Language Engineering
</SectionTitle>
      <Paragraph position="0"> Finally, this kind of in-depth corpus analysis gives us indications on what kind of syntactic annotation is needed ill order to lye able to use a corpus to perform tasks at the sentence level, and also, possibly, how to bootstrap a syntactic annotation process in a way that does not require much in-depth semantic knowledge about words.</Paragraph>
      <Paragraph position="1"> Had we wanted to imrform ~_he study reported in this paper by simple counting of occurrences in an appropriately annotated text -- thus eliminating the need tbr the tedious and time-consuming filtering of the automatic extraction which was necessary in the present study -- we would have needed a text annotated with categories deriw~d fl'mn knowledge about individual lexical items and a small portion of the tree surrounding them. First. all our comlts assumed knowledge of the verb classification in unergative, unaccusative and objectdrop, which requires mmotation of the thematic roles of the verb. Furthermore. tin&amp;quot; the cmmts of the several variables described we needed the verb items and the preceding auxiliary (activepassive and MV/p~st l~articiple), the following noun phrase and knowledge about whether the noun phrase was the direct object of the verb or not (transitive-intransitive). the preceding noun phrase and knowledge about whether the noun phrase wm~ the subject of the verb or rather an adjunct head (MV/reduced relative), an.d the preceding deterlniner (adjective-nonadjective). This is evidence in favour of annotation using a lexicalised formalism, whose main units are argument-structure dependencies between words, whether em:oded structurally, as in LTAG (Schabes and .loshi. 1991). or as grammatical relations, as in dclmmlency grammar (Hudson, 1990: Mel'cuk. 1988). From the point of view of parsing, these cmmts require only one chunk of text each.</Paragraph>
      <Paragraph position="2"> As an example, consider a grammatical formalisms, such as LTAG (Schabes and .loshi, 1991), which is both lexicalised and has been used to chunk text without pertbrnfing a fifll parse. An LTAG lexicon is a tbrest of lexicalised elementary trees. For verbs, the tree structure corresponds to their argument structure. Thus, each of the lexical items and portion of tree mentioned abow,, correspond to a different elementary tree. im:luding the unergative and unaccusative distinction, encoded by different labels referring to theinatic roles. Current  LTAG part-of-speech taggers, called supertaggets (Joshi and Srinivas, 1994; Srinivas, 1997) assign a set of elementary trees to each word, in effect chunking the text. The counts performed in the study reported here would have required simply counting the occurrences of the labels assigned to the words in the text by such a supertagger. Refinements in this direction of the annotation of the grammar used by the XTAG system (Doran et al., 1994) are actually tinder way.</Paragraph>
      <Paragraph position="3"> We also can see, from the raw frequencies obtained, that when collecting counts about syntactic phenomena, corpora must be in the order of hundreds of millions of words for the statistics to be reliable.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>