<?xml version="1.0" standalone="yes"?>
<Paper uid="P99-1069">
  <Title>Estimators for Stochastic &amp;quot;Unification-Based&amp;quot; Grammars*</Title>
  <Section position="3" start_page="0" end_page="536" type="metho">
    <SectionTitle>
2 Features in SUBGs
</SectionTitle>
    <Paragraph position="0"> We follow the statistical literature in using the term feature to refer to the properties that parameters are associated with (we use the word &amp;quot;attribute&amp;quot; to refer to the attributes or features of a UBG's feature structure). Let ~ be the set of all possible grammatical or well-formed analyses. Each feature f maps a syntactic analysis w E ~ to a real value f(w). The form of a syntactic analysis depends on the underlying linguistic theory. For example, for a PCFG w would be parse tree, for a LFG w would be a tuple consisting of (at least) a c-structure, an f-structure and a mapping from c-structure nodes to f-structure elements, and for a Chomskyian transformational grammar w would be a derivation. null Log-linear models are models in which the log probability is a linear combination of feature values (plus a constant). PCFGs, Gibbs distributions, Maximum-Entropy distributions and Markov Random Fields are all examples of log-linear models. A log-linear model associates each feature fj with a real-valued parameter Oj.</Paragraph>
    <Paragraph position="1">  A log-linear model with m features is one in which the likelihood P(w) of an analysis w is:</Paragraph>
    <Paragraph position="3"> While the estimators described below make no assumptions about the range of the .fi, in the models considered here the value of each feature fi(w) is the number of times a particular structural arrangement or configuration occurs in the analysis w, so fi(w) ranges over the natural numbers.</Paragraph>
    <Paragraph position="4"> For example, the features of a PCFG are indexed by productions, i.e., the value fi(w) of feature fi is the number of times the ith production is used in the derivation w.</Paragraph>
    <Paragraph position="5"> This set of features induces a tree-structured dependency graph on the productions which is characteristic of Markov Branching Processes (Pearl, 1988; Frey, 1998). This tree structure has the important consequence that simple &amp;quot;relative-frequencies&amp;quot; yield maximum-likelihood estimates for the Oi.</Paragraph>
    <Paragraph position="6"> Extending a PCFG model by adding additional features not associated with productions will in general add additional dependencies, destroy the tree structure, and substantially complicate maximum likelihood estimation.</Paragraph>
    <Paragraph position="7"> This is the situation for a SUBG, even if the features are production occurences. The unification constraints create non-local dependencies among the productions and the dependency graph of a SUBG is usually not a tree. Consequently, maximum likelihood estimation is no longer a simple matter of computing relative frequencies. But the resulting estimation procedures (discussed in detail, shortly), albeit more complicated, have the virtue of applying to essentially arbitrary features--of the production or non-production type. That is, since estimators capable of finding maximum-likelihood parameter estimates for production features in a SUBG will also find maximum-likelihood estimates for non-production features, there is no motivation for restricting features to be of the production type.</Paragraph>
    <Paragraph position="8"> Linguistically there is no particular reason for assuming that productions are the best features to use in a stochastic language model.</Paragraph>
    <Paragraph position="9"> For example, the adjunct attachment ambiguity in (1) results in alternative syntactic structures which use the same productions the same number of times in each derivation, so a model with only production features would necessarily assign them the same likelihood. Thus models that use production features alone predict that there should not be a systematic preference for one of these analyses over the other, contrary to standard psycholinguistic results.</Paragraph>
    <Paragraph position="10"> 1.a Bill thought Hillary \[vp\[vP left \] yesterday \] 1.b Bill \[vP\[vP thought Hillary left \] yesterday \] There are many different ways of choosing features for a SUBG, and each of these choices makes an empirical claim about possible distributions of sentences. Specifying the features of a SUBG is as much an empirical matter as specifying the grammar itself. For any given UBG there are a large (usually infinite) number of SUBGs that can be constructed from it, differing only in the features that each SUBG uses. In addition to production features, the stochastic LFG models evaluated below used the following kinds of features, guided by the principles proposed by Hobbs and Bear (1995).</Paragraph>
    <Paragraph position="11"> Adjunct and argument features indicate adjunct and argument attachment respectively, and permit the model to capture a general argument attachment preference. In addition, there are specialized adjunct and argument features corresponding to each grammatical function used in LFG (e.g., SUB J, OBJ, COMP, XCOMP, ADJUNCT, etc.). There are features indicating both high and low attachment (determined by the complexity of the phrase being attached to). Another feature indicates nonright-branching nonterminal nodes. There is a feature for non-parallel coordinate structures (where parallelism is measured in constituent structure terms). Each f-structure attributeatomic value pair which appears in any feature structure is also used as a feature. We also use a number of features identifying syntactic structures that seem particularly important in these corpora, such as a feature identifying NPs that are dates (it seems that date interpretations of NPs are preferred). We would have liked to have included features concerning specific lexical items (to capture head-to-head dependencies), but we felt that our corpora were so small  that the associated parameters could not be accurately estimated.</Paragraph>
  </Section>
  <Section position="4" start_page="536" end_page="538" type="metho">
    <SectionTitle>
3 A pseudo-likelihood estimator for
</SectionTitle>
    <Paragraph position="0"> log linear models Suppose ~ = Wl,..-,Wn is a training corpus of n syntactic analyses. Letting fj(~) = ~i=l,...,n fJ (wi), the log likelihood of the corpus and its derivatives are:</Paragraph>
    <Paragraph position="2"> where Eo(fj) is the expected value of fj under the distribution determined by the parameters 0. The maximum-likelihood estimates are the 0 which maximize log Lo(~). The chief difficulty in finding the maximum-likelihood estimates is calculating E0 (fj), which involves summing over the space of well-formed syntactic structures ft.</Paragraph>
    <Paragraph position="3"> There seems to be no analytic or efficient numerical way of doing this for a realistic SUBG.</Paragraph>
    <Paragraph position="4"> Abney (1997) proposes a gradient ascent, based upon a Monte Carlo procedure for estimating E0(fj). The idea is to generate random samples of feature structures from the distribution P~i(w), where 0 is the current parameter estimate, and to use these to estimate E~(fj), and hence the gradient of the likelihood. Samples are generated as follows: Given a SUBG, Abney constructs a covering PCFG based upon the SUBG and 0, the current estimate of 0. The derivation trees of the PCFG can be mapped onto a set containing all of the SUBG's syntactic analyses. Monte Carlo samples from the PCFG are comparatively easy to generate, and sample syntactic analyses that do not map to well-formed SUBG syntactic structures are then simply discarded. This generates a stream of syntactic structures, but not distributed according to P~(w) (distributed instead according to the restriction of the PCFG to the SUBG). Abney proposes using a Metropolis acceptancerejection method to adjust the distribution of this stream of feature structures to achieve detailed balance, which then produces a stream of feature structures distributed according to Po(w).</Paragraph>
    <Paragraph position="5"> While this scheme is theoretically sound, it would appear to be computationally impractical for realistic SUBGs. Every step of the proposed procedure (corresponding to a single step of gradient ascent) requires a very large number of PCFG samples: samples must be found that correspond to well-formed SUBGs; many such samples are required to bring the Metropolis algorithm to (near) equilibrium; many samples are needed at equilibrium to properly estimate E0(Ij).</Paragraph>
    <Paragraph position="6"> The idea of a gradient ascent of the likelihood (2) is appealing--a simple calculation reveals that the likelihood is concave and therefore free of local maxima. But the gradient (in particular, Ee(fj)) is intractable. This motivates an alternative strategy involving a data-based estimate of E0(fj):</Paragraph>
    <Paragraph position="8"> where y(w) is the yield belonging to the syntactic analysis w, and Yi = y(wi) is the yield belonging to the i'th sample in the training corpus. null The point is that Ee(fj(w)ly(w ) = Yi) is generally computable. In fact, if f~(y) is the set of well-formed syntactic structures that have yield y (i.e., the set of possible parses of the string y),</Paragraph>
    <Paragraph position="10"> Hence the calculation of the conditional expectations only involves summing over the possible syntactic analyses or parses f~(Yi) of the strings in the training corpus. While it is possible to construct UBGs for which the number of possible parses is unmanageably high, for many grammars it is quite manageable to enumerate the set of possible parses and thereby directly evaluate Eo(f j(w)ly(w ) = Yi).</Paragraph>
    <Paragraph position="11"> Therefore, we propose replacing the gradient,</Paragraph>
    <Paragraph position="13"> and performing a gradient ascent. Of course (6) is no longer the gradient of the likelihood func- null tion, but fortunately it is (exactly) the gradient of (the log of) another criterion:</Paragraph>
    <Paragraph position="15"> Instead of maximizing the likelihood of the syntactic analyses over the training corpus, we maximize the conditional likelihood of these analyses given the observed yields. In our experiments, we have used a conjugate-gradient optimization program adapted from the one presented in Press et al. (1992).</Paragraph>
    <Paragraph position="16"> Regardless of the pragmatic (computational) motivation, one could perhaps argue that the conditional probabilities Po(wly ) are as useful (if not more useful) as the full probabilities P0(w), at least in those cases for which the ultimate goal is syntactic analysis. Berger et al. (1996) and Jelinek (1997) make this same point and arrive at the same estimator, albeit through a maximum entropy argument.</Paragraph>
    <Paragraph position="17"> The problem of estimating parameters for log-linear models is not new. It is especially difficult in cases, such as ours, where a large sample space makes the direct computation of expectations infeasible. Many applications in spatial statistics, involving Markov random fields (MRF), are of this nature as well. In his seminal development of the MRF approach to spatial statistics, Besag introduced a &amp;quot;pseudolikelihood&amp;quot; estimator to address these difficulties (Besag, 1974; Besag, 1975), and in fact our proposal here is an instance of his method. In general, the likelihood function is replaced by a more manageable product of conditional likelihoods (a pseudo-likelihood--hence the designation PL0), which is then optimized over the parameter vector, instead of the likelihood itself.</Paragraph>
    <Paragraph position="18"> In many cases, as in our case here, this substitution side steps much of the computational burden without sacrificing consistency (more on this shortly).</Paragraph>
    <Paragraph position="19"> What are the asymptotics of optimizing a pseudo-likelihood function? Look first at the likelihood itself. For large n:</Paragraph>
    <Paragraph position="21"> where 0o is the true (and unknown) parameter vector. Up to a constant, (8) is the negative of the Kullback-Leibler divergence between the true and estimated distributions of syntactic analyses. As sample size grows, maximizing likelihood amofints to minimizing divergence.</Paragraph>
    <Paragraph position="22"> As for pseudo-likelihood:</Paragraph>
    <Paragraph position="24"> So that maximizing pseudo-likelihood (at large samples) amounts to minimizing the average (over yields) divergence between the true and estimated conditional distributions of analyses given yields.</Paragraph>
    <Paragraph position="25"> Maximum likelihood estimation is consistent: under broad conditions the sequence of distributions P0 , associated with the maximum r~ likelihood estimator for 0o given the samples Wl,...wn, converges to P0o. Pseudo-likelihood is also consistent, but in the present implementation it is consistent for the conditional distributions P0o (w\[y(w)) and not necessarily for the full distribution P0o (see Chi (1998)). It is not hard to see that pseudo-likelihood will not always correctly estimate P0o- Suppose there is a feature fi which depends only on yields: fi(w) = fi(y(w)). (Later we will refer to such features as pseudo-constant.) In this case, the derivative of PL0 (~) with respect to Oi is zero; PL0(~) contains no information about Oi. In fact, in this case any value of Oi gives the same conditional distribution Po(wly(w)); Oi is irrelevant to the problem of choosing good parses. Despite the assurance of consistency, pseudo-likelihood estimation is prone to over fitting when a large number of features is matched against a modest-sized training corpus. One particularly troublesome manifestation of over fitting results from the existence of features which, relative to the training set, we might term &amp;quot;pseudo-maximal&amp;quot;: Let us say that a feature f is pseudo-maximal for a yield y iff  Vw' E ~)(y)f(w) ~ f(J) where w is any correct parse of y, i.e., the feature's value on every correct parse w of y is greater than or equal to its value on any other parse of y. Pseudominimal features are defined similarly. It is easy to see that if fj is pseudo-maximal on each sentence of the training corpus then the parameter assignment Oj = co maximizes the corpus pseudo-likelihood. (Similarly, the assignment Oj = -oo maximizes pseudo-likelihood if fj is pseudo-minimal over the training corpus).</Paragraph>
    <Paragraph position="26"> Such infinite parameter values indicate that the model treats pseudo-maximal features categorically; i.e., any parse with a non-maximal feature value is assigned a zero conditional probability.</Paragraph>
    <Paragraph position="27"> Of course, a feature which is pseudo-maximal over the training corpus is not necessarily pseudo-maximal for all yields. This is an instance of over fitting, and it can be addressed, as is customary, by adding a regularization term that promotes small values of 0 to the objective function. A common choice is to add a quadratic to the log-likelihood, which corresponds to multiplying the likelihood itself by a normal distribution. In our experiments, we multiplied the pseudo-likelihood by a zero-mean normal in 01,... Om, with diagonal covariance, and with standard deviation aj for 0j equal to 7 times the maximum value of fj found in any parse in the training corpus. (We experimented with other values for aj, but the choice seems to have little effect). Thus instead of maximizing the log pseudo-likelihood, we choose 0 to maximize /3z 2</Paragraph>
    <Paragraph position="29"/>
  </Section>
  <Section position="5" start_page="538" end_page="538" type="metho">
    <SectionTitle>
4 A maximum correct estimator for
</SectionTitle>
    <Paragraph position="0"> log linear models The pseudo-likelihood estimator described in the last section finds parameter values which maximize the conditional probabilities of the observed parses (syntactic analyses) given the observed sentences (yields) in the training corpus. One of the empirical evaluation measures we use in the next section measures the number of correct parses selected from the set of all possible parses. This suggests another possible objective function: choose ~ to maximize the number Co (~) of times the maximum likelihood parse (under 0) is in fact the correct parse, in the training corpus.</Paragraph>
    <Paragraph position="1"> Co(~) is a highly discontinuous function of 0, and most conventional optimization algorithms perform poorly on it. We had the most success with a slightly modified version of the simulated annealing optimizer described in Press et al. (1992). This procedure is much more computationally intensive than the gradient-based pseudo-likelihood procedure. Its computational difficulty grows (and the quality of solutions degrade) rapidly with the number of features.</Paragraph>
  </Section>
  <Section position="6" start_page="538" end_page="539" type="metho">
    <SectionTitle>
5 Empirical evaluation
</SectionTitle>
    <Paragraph position="0"> Ron Kaplan and Hadar Shemtov at Xerox PArtC provided us with two LFG parsed corpora. The Verbmobil corpus contains appointment planning dialogs, while the Homecentre corpus is drawn from Xerox printer documentation. Table 1 summarizes the basic properties of these corpora. These corpora contain packed c/fstructure representations (Maxwell III and Kaplan, 1995) of the grammatical parses of each sentence with respect to Lexical-Functional grammars. The corpora also indicate which of these parses is in fact the correct parse (this information was manually entered). Because slightly different grammars were used for each corpus we chose not to combine the two corpora, although we used the set of features described in section 2 for both in the experiments described below. Table 2 describes the properties of the features used for each corpus.</Paragraph>
    <Paragraph position="1"> In addition to the two estimators described above we also present results from a baseline estimator in which all parses are treated as equally likely (this corresponds to setting all the parameters Oj to zero).</Paragraph>
    <Paragraph position="2"> We evaluated our estimators using held-out test corpus ~test. We used two evaluation measures. In an actual parsing application a SUBG might be used to identify the correct parse from the set of grammatical parses, so our first evaluation measure counts the number Co(~test) of sentences in the test corpus ~test whose maximum likelihood parse under the estimated model 0 is actually the correct parse.</Paragraph>
    <Paragraph position="3"> If a sentence has 1 most likely parses (i.e., all 1 parses have the same conditional probability) and one of these parses is the correct parse, then we score 1/l for this sentence.</Paragraph>
    <Paragraph position="4"> The second evaluation measure is the pseudo- null maximal and pseudo-minimal features do not include pseudo-constant features. likelihood itself, PL~(wtest). The pseudo-likelihood of the test corpus is the likelihood of the correct parses given their yields, so pseudo-likelihood measures how much of the probability mass the model puts onto the correct analyses. This metric seems more relevant to applications where the system needs to estimate how likely it is that the correct analysis lies in a certain set of possible parses; e.g., ambiguity-preserving translation and human-assisted disambiguation. To make the numbers more manageable, we actually present the negative logarithm of the pseudo-likelihood rather than the pseudo-likelihood itself--so smaller is better.</Paragraph>
    <Paragraph position="5"> Because of the small size of our corpora we evaluated our estimators using a 10-way cross-validation paradigm. We randomly assigned sentences of each corpus into 10 approximately equal-sized subcorpora, each of which was used in turn as the test corpus. We evaluated on each subcorpus the parameters that were estimated from the 9 remaining subcorpora that served as the training corpus for this run. The evaluation scores from each subcorpus were summed in order to provide the scores presented here.</Paragraph>
    <Paragraph position="6"> Table 3 presents the results of the empirical evaluation. The superior performance of both estimators on the Verbmobil corpus probably reflects the fact that the non-rule features were designed to match both the grammar and content of that corpus. The pseudo-likelihood estimator performed better than the correct-parses estimator on both corpora under both evaluation metrics. There seems to be substantial over learning in all these models; we routinely improved performance by discarding features. With a small number of features the correct-parses estimator typically scores better than the pseudo-likelihood estimator on the correct-parses evaluation metric, but the pseudo-likelihood estimator always scores better on the pseudo-likelihood evaluation metric. null</Paragraph>
  </Section>
class="xml-element"></Paper>