<?xml version="1.0" standalone="yes"?>
<Paper uid="P06-1124">
  <Title>Sydney, July 2006. c(c)2006 Association for Computational Linguistics A Hierarchical Bayesian Language Model based on Pitman-Yor Processes</Title>
  <Section position="4" start_page="985" end_page="986" type="metho">
    <SectionTitle>
2 Pitman-Yor Process
</SectionTitle>
    <Paragraph position="0"> Pitman-Yor processes are examples of nonparametric Bayesian models. Here we give a quick description of the Pitman-Yor process in the context of a unigram language model; good tutorials on such models are provided in (Ghahramani, 2005; Jordan, 2005). Let W be a fixed and finite vocabulary of V words. For each word w[?]W let G(w) be the (to be estimated) probability of w, and let G = [G(w)]w[?]W be the vector of word probabilities. We place a Pitman-Yor process prior on G:</Paragraph>
    <Paragraph position="2"> where the three parameters are: a discount parameter 0[?]d &lt; 1, a strength parameter th &gt;[?]d and a mean vector G0 = [G0(w)]w[?]W . G0(w) is the a priori probability of word w: before observing any data, we believe word w should occur with probability G0(w). In practice this is usually set uniformly G0(w) = 1/V for all w [?]W. Both th and d can be understood as controlling the amount of variability around G0 in different ways. When d = 0 the Pitman-Yor process reduces to a Dirichlet distribution with parameters thG0.</Paragraph>
    <Paragraph position="3"> There is in general no known analytic form for the density of PY(d,th,G0) when the vocabulary is finite. However this need not deter us as we will instead work with the distribution over sequences of words induced by the Pitman-Yor process, which has a nice tractable form and is sufficient for our purpose of language modelling. To be precise, notice that we can treat both G and G0 as distributions over W, where word w [?] W has probability G(w) (respectively G0(w)). Let x1,x2,... be a sequence of words drawn independently and identically (i.i.d.) from G. We shall describe the Pitman-Yor process in terms of a generative procedure that produces x1,x2,... iteratively with G marginalized out. This can be achieved by relating x1,x2,... to a separate sequence of i.i.d. draws y1,y2,... from the mean distribution G0 as follows. The first word x1 is assigned the value of the first draw y1 from G0.</Paragraph>
    <Paragraph position="4"> Let t be the current number of draws from G0 (currently t = 1), ck be the number of words assigned the value of draw yk (currently c1 = 1), and c* =summationtexttk=1 ck be the current number of draws from G. For each subsequent word xc*+1, we either assign it the value of a previous draw yk with probability ck[?]dth+c* (increment ck; set xc*+1 -yk), or we assign it the value of a new draw from G0  scale, with d = .5 and th = 1 (bottom), 10 (middle) and 100 (top). Second panel: same, with th = 10 and d = 0 (bottom), .5 (middle) and .9 (top). Third panel: proportion of words appearing only once, as a function of the number of words drawn, with d = .5 and th = 1 (bottom), 10 (middle), 100 (top). Last panel: same, with th = 10 and d = 0 (bottom), .5 (middle) and .9 (top). with probability th+dtth+c* (increment t; set ct = 1; draw yt[?]G0; set xc*+1-yt).</Paragraph>
    <Paragraph position="5"> The above generative procedure produces a sequence of words drawn i.i.d. from G, with G marginalized out. It is informative to study the Pitman-Yor process in terms of the behaviour it induces on this sequence of words. Firstly, notice the rich-gets-richer clustering property: the more words have been assigned to a draw from G0, the more likely subsequent words will be assigned to the draw. Secondly, the more we draw from G0, the more likely a new word will be assigned to a new draw from G0. These two effects together produce a power-law distribution where many unique words are observed, most of them rarely. In particular, for a vocabulary of unbounded size and for d &gt; 0, the number of unique words scales as O(thTd) where T is the total number of words. For d = 0, we have a Dirichlet distribution and the number of unique words grows more slowly at O(thlogT).</Paragraph>
    <Paragraph position="6"> Figure 1 demonstrates the power-law behaviour of the Pitman-Yor process and how this depends on d and th. In the first two panels we show the average number of unique words among 10 sequences of T words drawn from G, as a function of T, for various values of th and d. We see that th controls the overall number of unique words, while d controls the asymptotic growth of the number of unique words. In the last two panels, we show the proportion of words appearing only once among the unique words; this gives an indication of the proportion of words that occur rarely. We see that the asymptotic behaviour depends on d but not on th, with larger d's producing more rare words.</Paragraph>
    <Paragraph position="7"> This procedure for generating words drawn from G is often referred to as the Chinese restaurant process (Pitman, 2002). The metaphor is as follows. Consider a sequence of customers (corresponding to the words draws from G) visiting a Chinese restaurant with an unbounded number of tables (corresponding to the draws from G0), each of which can accommodate an unbounded number of customers. The first customer sits at the first table, and each subsequent customer either joins an already occupied table (assign the word to the corresponding draw from G0), or sits at a new table (assign the word to a new draw from G0).</Paragraph>
  </Section>
  <Section position="5" start_page="986" end_page="987" type="metho">
    <SectionTitle>
3 Hierarchical Pitman-Yor Language
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="986" end_page="987" type="sub_section">
      <SectionTitle>
Models
</SectionTitle>
      <Paragraph position="0"> We describe an n-gram language model based on a hierarchical extension of the Pitman-Yor process.</Paragraph>
      <Paragraph position="1"> An n-gram language model defines probabilities over the current word given various contexts consisting of up to n[?]1 words. Given a context u, let Gu(w) be the probability of the current word taking on value w. We use a Pitman-Yor process as the prior for Gu[Gu(w)]w[?]W , in particular,</Paragraph>
      <Paragraph position="3"> where pi(u) is the suffix of u consisting of all but the earliest word. The strength and discount parameters are functions of the length|u|of the context, while the mean vector is Gpi(u), the vector of probabilities of the current word given all but the earliest word in the context. Since we do not know Gpi(u) either, We recursively place a prior over Gpi(u) using (3), but now with parameters th|pi(u)|,d|pi(u) |and mean vector Gpi(pi(u)) instead.</Paragraph>
      <Paragraph position="4"> This is repeated until we get to G[?], the vector of probabilities over the current word given the  empty context[?]. Finally we place a prior on G[?]:</Paragraph>
      <Paragraph position="6"> where G0 is the global mean vector, given a uniform value of G0(w) = 1/V for all w [?]W. Finally, we place a uniform prior on the discount parameters and a Gamma(1,1) prior on the strength parameters. The total number of parameters in the model is 2n.</Paragraph>
      <Paragraph position="7"> The structure of the prior is that of a suffix tree of depth n, where each node corresponds to a context consisting of up to n[?]1 words, and each child corresponds to adding a different word to the beginning of the context. This choice of the prior structure expresses our belief that words appearing earlier in a context have (a priori) the least importance in modelling the probability of the current word, which is why they are dropped first at successively higher levels of the model.</Paragraph>
    </Section>
  </Section>
  <Section position="6" start_page="987" end_page="988" type="metho">
    <SectionTitle>
4 Hierarchical Chinese Restaurant
</SectionTitle>
    <Paragraph position="0"> We describe a generative procedure analogous to the Chinese restaurant process of Section 2 for drawing words from the hierarchical Pitman-Yor language model with all Gu's marginalized out. This gives us an alternative representation of the hierarchical Pitman-Yor language model that is amenable to efficient inference using Markov chain Monte Carlo sampling and easy computation of the predictive probabilities for test words. The correspondence between interpolated Kneser-Ney and the hierarchical Pitman-Yor language model is also apparent in this representation.</Paragraph>
    <Paragraph position="1"> Again we may treat each Gu as a distribution over the current word. The basic observation is that since Gu is Pitman-Yor process distributed, we can draw words from it using the Chinese restaurant process given in Section 2. Further, the only operation we need of its parent distribution Gpi(u) is to draw words from it too. Since Gpi(u) is itself distributed according to a Pitman-Yor process, we can use another Chinese restaurant process to draw words from that. This is recursively applied until we need draws from the global mean distribution G0, which is easy since it is just uniform. We refer to this as the hierarchical Chinese restaurant process.</Paragraph>
    <Paragraph position="2"> Let us introduce some notations. For each context u we have a sequence of words xu1,xu2,...</Paragraph>
    <Paragraph position="3"> drawn i.i.d. from Gu and another sequence of words yu1,yu2,... drawn i.i.d. from the parent distribution Gpi(u). We use l to index draws from Gu and k to index the draws from Gpi(u). Define tuwk = 1 if yuk takes on value w, and tuwk = 0 otherwise. Each word xul is assigned to one of the draws yuk from Gpi(u). If yuk takes on value w define cuwk as the number of words xul drawn from Gu assigned to yuk, otherwise let cuwk = 0.</Paragraph>
    <Paragraph position="4"> Finally we denote marginal counts by dots. For example, cu*k is the number of xul's assigned the value of yuk, cuw* is the number of xul's with value w, and tu** is the current number of draws yuk from Gpi(u). Notice that we have the following relationships among the cuw*'s and tuw*:</Paragraph>
    <Paragraph position="6"> Pseudo-code for drawing words using the hierarchical Chinese restaurant process is given as a recursive function DrawWord(u), while pseudo-code for computing the probability that the next word drawn from Gu will be w is given in WordProb(u,w). The counts are initialized at all</Paragraph>
    <Paragraph position="8"> Returns a new word drawn from Gu.</Paragraph>
    <Paragraph position="9"> If u = 0, return w[?]W with probability G0(w). Else with probabilities proportional to: cuwk[?]d|u|tuwk: assign the new word to yuk.</Paragraph>
    <Paragraph position="10"> Increment cuwk; return w.</Paragraph>
    <Paragraph position="11"> th|u |+ d|u|tu**: assign the new word to a new draw yuknew from Gpi(u).</Paragraph>
    <Paragraph position="12"> Let w-DrawWord(pi(u)); set tuwknew = cuwknew = 1; return w.</Paragraph>
    <Paragraph position="13"> Function WordProb(u,w): Returns the probability that the next word after context u will be w.</Paragraph>
    <Paragraph position="15"> Notice the self-reinforcing property of the hierarchical Pitman-Yor language model: the more a word w has been drawn in context u, the more likely will we draw w again in context u. In fact word w will be reinforced for other contexts that share a common suffix with u, with the probability of drawing w increasing as the length of the  common suffix increases. This is because w will be more likely under the context of the common suffix as well.</Paragraph>
    <Paragraph position="16"> The hierarchical Chinese restaurant process is equivalent to the hierarchical Pitman-Yor language model insofar as the distribution induced on words drawn from them are exactly equal. However, the probability vectors Gu's have been marginalized out in the procedure, replaced instead by the assignments of words xul to draws yuk from the parent distribution, i.e. the seating arrangement of customers around tables in the Chinese restaurant process corresponding to Gu. In the next section we derive tractable inference schemes for the hierarchical Pitman-Yor language model based on these seating arrangements.</Paragraph>
  </Section>
  <Section position="7" start_page="988" end_page="989" type="metho">
    <SectionTitle>
5 Inference Schemes
</SectionTitle>
    <Paragraph position="0"> In this section we give a high level description of a Markov chain Monte Carlo sampling based inference scheme for the hierarchical Pitman-Yor language model. Further details can be obtained at (Teh, 2006). We also relate interpolated Kneser-Ney to the hierarchical Pitman-Yor language model.</Paragraph>
    <Paragraph position="1"> Our training data D consists of the number of occurrences cuw* of each word w after each context u of length exactly n[?]1. This corresponds to observing word w drawn cuw* times from Gu.</Paragraph>
    <Paragraph position="2"> Given the training data D, we are interested in the posterior distribution over the latent vectors</Paragraph>
    <Paragraph position="4"> As mentioned previously, the hierarchical Chinese restaurant process marginalizes out each Gu, replacing it with the seating arrangement in the corresponding restaurant, which we shall denote by Su. Let S = {Sv : all contexts v}. We are thus interested in the equivalent posterior over seating arrangements instead:</Paragraph>
    <Paragraph position="6"> The most important quantities we need for language modelling are the predictive probabilities: what is the probability of a test word w after a context u? This is given by</Paragraph>
    <Paragraph position="8"> where the first probability on the right is the predictive probability under a particular setting of seating arrangementsSand parameters Th, and the overall predictive probability is obtained by averaging this with respect to the posterior overS and Th (second probability on right). We approximate the integral with samples {S(i),Th(i)}Ii=1 drawn from p(S,Th|D):</Paragraph>
    <Paragraph position="10"> where the counts are obtained from the seating arrangement Su in the Chinese restaurant process corresponding to Gu.</Paragraph>
    <Paragraph position="11"> We use Gibbs sampling to obtain the posterior samples {S,Th} (Neal, 1993). Gibbs sampling keeps track of the current state of each variable of interest in the model, and iteratively resamples the state of each variable given the current states of all other variables. It can be shown that the states of variables will converge to the required samples from the posterior distribution after a sufficient number of iterations. Specifically for the hierarchical Pitman-Yor language model, the variables consist of, for each u and each word xul drawn from Gu, the index kul of the draw from Gpi(u) assigned xul. In the Chinese restaurant metaphor, this is the index of the table which the lth customer sat at in the restaurant corresponding to Gu. If xul has value w, it can only be assigned to draws from Gpi(u) that has value w as well. This can either be a preexisting draw with value w, or it can be a new draw taking on value w. The relevant probabilities are given in the functions DrawWord(u) and WordProb(u,w), where we treat xul as the last word drawn from Gu. This gives:</Paragraph>
    <Paragraph position="13"> where the superscript [?]ul means the corresponding set of variables or counts with xul excluded.</Paragraph>
    <Paragraph position="14"> The parameters Th are sampled using an auxiliary variable sampler as detailed in (Teh, 2006). The overall sampling scheme for an n-gram hierarchical Pitman-Yor language model takes O(nT) time and requires O(M) space per iteration, where T is the number of words in the training set, and M is the number of unique n-grams. During test time, the computational cost is O(nI), since the predictive probabilities (12) require O(n) time to calculate for each of I samples.</Paragraph>
    <Paragraph position="15"> The hierarchical Pitman-Yor language model produces discounts that grow gradually as a function of n-gram counts. Notice that although each Pitman-Yor process Gu only has one discount parameter, the predictive probabilities (12) produce different discount values since tuw* can take on different values for different words w. In fact tuw* will on average be larger if cuw* is larger; averaged over the posterior, the actual amount of discount will grow slowly as the count cuw* grows. This is shown in Figure 2 (left), where we see that the growth of discounts is sublinear.</Paragraph>
    <Paragraph position="16"> The correspondence to interpolated Kneser-Ney is now straightforward. If we restrict tuw* to be at most 1, that is,</Paragraph>
    <Paragraph position="18"> we will get the same discount value so long as cuw* &gt; 0, i.e. absolute discounting. Further supposing that the strength parameters are all th|u |= 0, the predictive probabilities (12) now directly reduces to the predictive probabilities given by interpolated Kneser-Ney. Thus we can interpret interpolated Kneser-Ney as the approximate inference scheme (15,16) in the hierarchical Pitman-Yor language model.</Paragraph>
    <Paragraph position="19"> Modified Kneser-Ney uses the same values for the counts as in (15,16), but uses a different valued discount for each value of cuw* up to a maximum of c(max). Since the discounts in a hierarchical Pitman-Yor language model are limited to between 0 and 1, we see that modified Kneser-Ney is not an approximation of the hierarchical Pitman-Yor language model.</Paragraph>
  </Section>
  <Section position="8" start_page="989" end_page="990" type="metho">
    <SectionTitle>
6 Experimental Results
</SectionTitle>
    <Paragraph position="0"> We performed experiments on the hierarchical Pitman-Yor language model on a 16 million word corpus derived from APNews. This is the same dataset as in (Bengio et al., 2003). The training, validation and test sets consist of about 14 million, 1 million and 1 million words respectively, while the vocabulary size is 17964. For trigrams with n = 3, we varied the training set size between approximately 2 million and 14 million words by six equal increments, while we also experimented with n = 2 and 4 on the full 14 million word training set. We compared the hierarchical Pitman-Yor language model trained using the proposed Gibbs sampler (HPYLM) against interpolated Kneser-Ney (IKN), modified Kneser-Ney (MKN) with maximum discount cut-off c(max) = 3 as recommended in (Chen and Goodman, 1998), and the hierarchical Dirichlet language model (HDLM).</Paragraph>
    <Paragraph position="1"> For the various variants of Kneser-Ney, we first determined the parameters by conjugate gradient descent in the cross-entropy on the validation set.</Paragraph>
    <Paragraph position="2"> At the optimal values, we folded the validation set into the training set to obtain the final n-gram probability estimates. This procedure is as recommended in (Chen and Goodman, 1998), and takes approximately 10 minutes on the full training set with n = 3 on a 1.4 Ghz PIII. For HPYLM we inferred the posterior distribution over the latent variables and parameters given both the training and validation sets using the proposed Gibbs sampler. Since the posterior is well-behaved and the sampler converges quickly, we only used 125 iterations for burn-in, and 175 iterations to collect posterior samples. On the full training set with n = 3 this took about 1.5 hours.</Paragraph>
    <Paragraph position="3"> Perplexities on the test set are given in Table 1.</Paragraph>
    <Paragraph position="4"> As expected, HDLM gives the worst performance, while HPYLM performs better than IKN. Perhaps surprisingly HPYLM performs slightly worse than MKN. We believe this is because HPYLM is not a perfect model for languages and as a result posterior estimates of the parameters are not optimized for predictive performance. On the other hand parameters in the Kneser-Ney variants are optimized using cross-validation, so are given optimal values for prediction. To validate this conjecture, we also experimented with HPYCV, a hierarchical Pitman-Yor language model where the parameters are obtained by fitting them in a slight generalization of IKN where the strength param- null various sizes of training set T and length of ngrams. null eters th|u|'s are allowed to be positive and optimized over along with the discount parameters using cross-validation. Seating arrangements are Gibbs sampled as in Section 5 with the parameter values fixed. We find that HPYCV performs better than MKN (except marginally worse on small problems), and has best performance overall. Note that the parameter values in HPYCV are still not the optimal ones since they are obtained by cross-validation using IKN, an approximation to a hierarchical Pitman-Yor language model. Unfortunately cross-validation using a hierarchical Pitman-Yor language model inferred using Gibbs sampling is currently too costly to be practical.</Paragraph>
    <Paragraph position="5"> In Figure 2 (right) we broke down the contributions to the cross-entropies in terms of how many times each word appears in the test set. We see that most of the differences between the methods appear as differences among rare words, with the contribution of more common words being negligible. HPYLM performs worse than MKN on words that occurred only once (on average) and better on other words, while HPYCV is reversed and performs better than MKN on words that occurred only once or twice and worse on other words.</Paragraph>
  </Section>
class="xml-element"></Paper>