<?xml version="1.0" standalone="yes"?>
<Paper uid="P06-2065">
  <Title>Unsupervised Analysis for Decipherment Problems</Title>
  <Section position="5" start_page="0" end_page="500" type="metho">
    <SectionTitle>
3 English Letter Substitution
</SectionTitle>
    <Paragraph position="0"> An informal substitution cipher (Smith, 1943) disguises a text by substituting code letters for normal letters. This system is usually exclusive, meaning that each plaintext letter maps to only one ciphertext letter, and vice versa. There is surprisingly little published on this problem, e.g., (Peleg and Rosenfeld, 1979), because fast computers led to public-key cryptography before much computer analysis was done on such old-style ciphers. We study this problem rst because it resembles many of the other problems we are interested in, and we can generate arbitrary amounts of test data.</Paragraph>
    <Paragraph position="1"> We estimate unsmoothed parameter values for an English letter-bigram P(p) from news data. This is a 27x27 table that includes the space character. We then set up a uniform P(c  |p), which also happens to be a  (a) ingcmpnqsnwf cv fpn owoktvcv hu ihgzsnwfv rqcffnw cw owgcnwf kowazoanv... (b) wecitherkent is the analysis of wocoments pritten in ancient buncquges... (c) decipherment is the analysis of documents written in ancient languages...  improved decipherment.</Paragraph>
    <Paragraph position="2"> Given a ciphertext c of length a2 , a plaintext vocabulary of a3 tokens, and a plaintext bigram model b:  1. set a s(a4a6a5a7 ) substitution table initially to be uniform 2. for several iterations do: a. set up a count table count(a4 , a7 ) with zero entries b. P(c) = 0 c. for all possible plaintexts a7a9a8a11a10a12a10a13a10a14a7a16a15 (each a7a16a17 drawn from plaintext vocabulary)</Paragraph>
    <Paragraph position="4"> other values to 1/26. We create our ciphertext by encrypting an out-of-domain encyclopedia article. This article contains 417 letters, some of which are shown in Figure 3(a).</Paragraph>
    <Paragraph position="5"> The decipherment yielded by EM/Viterbi contains 68 errors see Figure 3(b).</Paragraph>
    <Paragraph position="6"> Can we do better? First, we are not taking advantage of the fact that the cipher system is exclusive. But, as we observe in the rest of this paper, most natural decipherment problems do not have this feature, so we do not take advantage of it in this case (and it is hard to model!).</Paragraph>
    <Paragraph position="7"> We can certainly acquire vastly more data for estimating P(p). Using a 1.5-million character data set instead of a 70,000-character data set reduces the number of errors from 68 to 64. Next, we apply xed-lambda interpolation smoothing to P(p). This reduces errors further to 62.</Paragraph>
    <Paragraph position="8"> Next, we adjust our Viterbi search to maximize P(p) a1 P(c  |p)a36 rather than P(p) a1 P(c  |p). This cubing concept was introduced in another context by (Knight and Yamada, 1999). It serves to stretch out the P(c  |p) probabilities, which tend to be too bunched up. This bunching is caused by incompatibilities between the n-gram frequencies used to train P(p) and the n-gram frequencies found in the correct decipherment of c. We nd this technique extremely useful across decipherment applications. Here it reduces errors from 62 down to 42.</Paragraph>
    <Paragraph position="9"> We also gain by using letter trigrams instead of bi-Given a ciphertext c of length a2 , a plaintext vocabulary of a3 tokens, and a plaintext bigram model b:  1. set the s(a4a31a5a7 ) substitution table initially to be uniform 2. for several iterations do: a. set up a count(a4 , a7 ) table with zero entries</Paragraph>
    <Paragraph position="11"> g. normalize count(a4 , a7 ) table to create a revised s(a4a31a5a7 )  plishes the same thing as Figure 1.</Paragraph>
    <Paragraph position="12"> grams. This reduces error from the original 68 to 57 (small source data) or 32 (large source data). Combining trigrams with cubing the channel probabilities reduces error to 15, which source-model smoothing further reduces to 10 (or 2.4%), as in Figure 3(c).</Paragraph>
    <Paragraph position="13"> So far we have glossed over the number of EM iterations used. From the EM's point of view, the more iterations, the better, as these improve P(c). However, the decipherment error rate may jump around as iterations proceed. Figure 4 shows the effect of EM iterations on error rate. With the worse source models, it is better to stop the EM early. EM initially locks onto the correct theory, but task performance degrades as it tries to make the ciphertext decoding t the expected bigram frequencies. Better source models do not suffer much.</Paragraph>
    <Paragraph position="14"> If we give the system more knowledge about English vocabulary and grammar, it will further improve. We have also been able to get perfect performance by using the best-so-far decipherment in Figure 3 to pull down related English texts from the web, and using these to retrain P(p) to fuel a second decipherment. However, we only present the simple substitution cipher as a prototype of the kinds of applications we are really interested in, which we present in the following sections. The experiments we have presented so far should not be viewed as tuning parameters for performance  indeed, it is not correct to measure accuracy on a tuning/development data set. Rather, we have demonstrated some general strategies and observations (more data, larger n-grams, stability of good language models) that we can apply to other real decipherment situations. In many such situations, there is only a test set, and tuning is impossible even in principle fortunately, we observe that the general strategies work robustly across a number of decipherment domains.</Paragraph>
  </Section>
  <Section position="6" start_page="500" end_page="501" type="metho">
    <SectionTitle>
4 Character Code Conversion
</SectionTitle>
    <Paragraph position="0"> Many human languages are straightforwardly represented at the character level by some widely-adopted standard (e.g., ASCII). In dealing with other languages (like Arabic), we must be equally prepared to process a few different standards. Documents in yet other languages (like Hindi) are found spread across the web in dozens if not hundreds of specialized encodings. These come with downloadable fonts for viewing. However, they are dif cult to handle by computer, for example, to build a full-coverage Hindi web-search engine, or to pool Hindi corpora for training machine translation or speech recognition.</Paragraph>
    <Paragraph position="1"> Character conversion tools exist for many pairs of major encoding systems, but it has been the experience of many researchers that these tools are awed, despite the amount of work that goes into them. 100% accuracy is not to be found. Furthermore, nothing exists for most pairs. We believe that mild annotation techniques allow people to generate conversion tables quite quickly (and we show some results on this), but we follow here an unsupervised approach, as would be required to automatically generate a consistentlyencoded Hindi web.</Paragraph>
    <Paragraph position="2"> Our ciphertext c is a stream of bytes in an unknown encoding, with space separators; we use integers to represent these bytes, as in Figure 5(a). Our plaintext is a large collection of UTF8 standard Hindi. UTF8 builds complex Hindi character chunks out of up to 3 simple and combining characters. A Hindi word is a sequence of chunks, and words are separated by spaces.</Paragraph>
    <Paragraph position="3"> We know that c is Hindi we imagine that it was once UTF8, but that it somehow got enciphered.</Paragraph>
    <Paragraph position="4"> Modeling is more complex than in the previous section. First, we have to decide what our plaintext tokens will be. Our rst approach was to use chunks. Chunk boundaries are essentially those where we could draw a vertical line in written Hindi without disturbing any characters. We could then set up a model of how UTF8 is encoded to the mystery sequence in the putative channel namely, we let each source chunk map to a particular target byte sequence. (By analogy, we would divide up English text into mostly letters, but would chunk ligatures like together. In fact, in extracting English text from pdf, we often nd encoded by a single byte). This model is quite general and holds up across the encodings we have dealt with. However, there are over 900 chunks to contend with, and vast numbers of target byte sequences, so that the P(c  |p) table is nearly unmanageable.</Paragraph>
    <Paragraph position="5"> Therefore, we use a simpler model. We divide p into individual characters, and we set up a channel in which plaintext characters can map into either one or two ciphertext bytes. Instead of a table like P(c c  |p), we set up two tables: P(f  |p) for character fertility, and P(c  |p) for character-to-byte substitution. This is similar to Model 3 of (Brown et al., 1993), but without null-generated elements or re-ordering.</Paragraph>
    <Paragraph position="6"> Our actual ciphertext is an out-of-domain web page with 11,917 words of song lyrics in Hindi, in an idiosyncratic encoding. There is no known tool to convert from this encoding. In order to report error rates, we had to manually annotate a portion of this web page with correct UTF8. This was quite dif cult. We were completely unable to do this manually by relying only on the ciphertext byte sequence even though this is what we are asking our machine to do! But as Hindi readers, we also have access to the web-site rendering in Hindi glyphs, which helps us identify which byte sequences correspond to which Hindi glyphs, and then to UTF8. The labeled portion of our ciphertext consists of 59 running words (281 ciphertext bytes and 201 UTF8 characters).</Paragraph>
    <Paragraph position="7"> Because the machine decipherment rarely consists of exactly 201 UTF8 characters, we report edit distance instead of error rate. An edit distance of 0 is perfect, while the edit distance for long incorrect decipherments may be greater than 201. With a source character bi-gram model, and the above channel, we obtain an edit distance of 161. With a trigram model, we get 127.</Paragraph>
    <Paragraph position="8"> Now we introduce another idea that has worked across several decipherment problems. We use a xed, uniform fertility model and allow EM only to manip- null (a) ... 13 5 14 . 16 2 25 26 2 25 . 17 2 13 . 15 2 8 . 7 2 4 2 9 2 2 ... (b) ... 6 35 . 12 28 49 10 28 . 3 4 6 . 1 10 3 . 29 4 8 20 4 ... (c) ... 6 35 24 . 12 28 21 4 . 11 6 . 12 25 . 29 8 22 4 ... (d) ... 6/35/24 . 12/28 21/28 . 3/4 6 . 1/25 . 29 8 20/4 ... *</Paragraph>
    <Paragraph position="10"> probabilities for Hindi decipherment. Correct mappings are marked with *.</Paragraph>
    <Paragraph position="11"> ulate substitution probabilities. This prevents the algorithm from locking onto bad solutions. This gives an improved solution edit distance of 93, as in Figure 5(b), which can be compared to the correct decipherment in 5(d). Figure 6 shows a portion of the learned P(c  |p) substitution table, with * indicating correct mappings.</Paragraph>
    <Paragraph position="12"> 15 out of 59 test words are deciphered exactly correctly. Another 16 out of 59 are perfect except for the addition of one extra UTF8 character (always 4 or 25 ). Ours are the rst results we know of with unsupervised techniques.</Paragraph>
    <Paragraph position="13"> We also experimented with using a word-based source model in place of the character n-gram model.</Paragraph>
    <Paragraph position="14"> We built a word-unigram P(p) model out of only the top 5000 UTF8 words in our source corpus it assigns probability zero to any word not in this list. This is a harsh model, considering that 16 out of 59 words in our UTF8-annotated test corpus do not even occur in the list, and are thus unreachable. On the plus side, EM considers only decipherments consisting of sequences of real Hindi words, and the Viterbi decoder only generates genuine Hindi words. The resulting decipherment edit distance is encouraging at 92, with the result shown in Figure 5(c). This model correctly deciphers 25 out of 59 words, with only some overlap to the previous 15 correct out of 59 one or other of the models is able to perfectly decipher 31 out of 59 words already, making a combination promising.</Paragraph>
    <Paragraph position="15"> Our machine is also able to learn in a semi-supervised manner by aligning a cipher corpus with a manually-done translation into UTF8. EM searches for the parameter settings that maximize P(c  |p), and a Viterbi alignment is a by-product. For the intuition, see Figure 5(a and d), in which plaintext character 6 occurs twice and may be guessed to correspond with ciphertext byte 13 . EM does this perfectly, except for some regions where re-ordering indeed happens.</Paragraph>
    <Paragraph position="16"> We are able to move back to our chunk-based model in semi-supervised mode, which avoids the re-ordering problem, and we obtain near-perfect decipherment tables when we asked a human to re-type a few hundred words of mystery-encoded text in a UTF8 editor.</Paragraph>
  </Section>
  <Section position="7" start_page="501" end_page="502" type="metho">
    <SectionTitle>
5 Phonetic Decipherment
</SectionTitle>
    <Paragraph position="0"> This section expands previous work on phonetic decipherment (Knight and Yamada, 1999). Archaeologists are often faced with an unknown writing system that is believed to represent a known spoken language. That is, the written characters encode phonetic sequences (sometimes individual phonemes, and sometimes whole words), and the relationship between text and sound is to be discovered, followed by the meaning. Viewing text as a code for speech was radical some years ago. It is now the standard view of writing systems, and many even view written Chinese as a straightforward syllabary, albeit one that is much larger and complex than, say, Japanese kana. Both Linear B and Mayan writing were deciphered by viewing the observed text as a code/cipher for an approximatelyknown spoken language (Chadwick, 1958; Coe, 1993). We follow (Knight and Yamada, 1999) in using Spanish as an example. The ciphertext is a 6980character passage from Don Quixote, as in Figure 7(a).</Paragraph>
    <Paragraph position="1"> The plaintext is a very large out-of-domain Spanish phoneme sequence from which we compute only phoneme n-gram probabilities. We try deciphering without detailed knowledge of spoken Spanish words and grammar. The goal is for the decipherment to be understandable by modern Spanish speakers.</Paragraph>
    <Paragraph position="2"> First, it is necessary to settle on the basic inventory of sounds and characters. Characters are easy; we simply tabulate the distinct ones observed in ciphertext.</Paragraph>
    <Paragraph position="3"> For sounds, we use a Spanish-relevant subset of the International Phonetic Alphabet (IPA), which seeks to capture all sounds in all languages; the implementation is SAMPA (Speech Assessment Methods Phonetic Alphabet). Here we show the sound and character inventories: null Sounds: B, D, G, J (ny as in canyon), L (y as in yarn), T (th as in thin), a, b, d, e, f, g, i, k, l, m, n, o, p, r, rr (trilled), s, t, tS (ch as in chin), u, x (h as in hat)  (a) primera parte del ingenioso hidalgo don quijote de la mancha (b) primera parte des intenioso liDasto don fuiLote de la manTia (c) primera parte del inGenioso biDalGo don fuiLote de la manTia (d) primera parte del inxenioso iDalGo don kixote de la manSa *  decipherment, and (d) is the correct phonetic transcription. Characters: ae, AE, O, , , oe, a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t, u, v, w, x, y, z The correct decipherment (Figure 7(d)) is a sequence of 6759 phonemes (here in SAMPA IPA).</Paragraph>
    <Paragraph position="4"> We use a P(c  |p) model that substitutes a single letter for each phoneme throughout the sequence. This considerably violates the rules of written Spanish (e.g., the K sound is often written with two letters q u, and the two K S sounds are often written x), so we do not expect a perfect decipherment. We do not enforce exclusivity; for example, the S sound may be written as c or s.</Paragraph>
    <Paragraph position="5"> An unsmoothed phonetic bigram model gives an edit distance (error) of 805, as in Figure 7(b). Here we study smoothing techniques. A xed-lambda interpolation smoothing yields 684 errors, while giving each phoneme its own trainable lambda yields a further reduction to 621. The corresponding edit distances for a trigram source model are 595, 703, and 492, the latter shown in Figure 7(c), an error of 7%. (This result is equivalent to Knight &amp; Yamada [1999]'s 4% error, which did not count extra incorrect phonemes produced by decipherment, such as pronunciations of silent letters). Quality smoothing yields the best results. While even the best decipherment is awed, it is perfectly understandable when synthesized, and it is very good with respect to the structure of the channel model.</Paragraph>
  </Section>
  <Section position="8" start_page="502" end_page="503" type="metho">
    <SectionTitle>
6 Universal Phonetic Decipherment
</SectionTitle>
    <Paragraph position="0"> What if the language behind the script is unknown? The next two sections address this question in two different ways.</Paragraph>
    <Paragraph position="1"> One idea is to look for universal constraints on phoneme sequences. If we somehow know that P(K AE N UW L IY) is high, while P(R T M K T K) is low, that we may be able to exploit such knowledge in deciphering an alphabetic writing system. In fact, many universal constraints have been proposed by linguists. Two major camps include syllable theorists (who say that words are composed of syllables, and syllables have internal regular structure (Blevins, 1995)) and anti-syllable theorists (who say that words are composed of phonemes that often constrain each other even across putative syllable boundaries (Steriade, 1998)).</Paragraph>
    <Paragraph position="2"> We use the same Don Quixote ciphertext as in the previous section. While the ultimate goal is to label each letter with a phoneme, we rst attack a more tractable problem, that of labeling each letter as C (consonant) or V (vowel). Once we know which letters stand for consonant sounds, we can break them down further.</Paragraph>
    <Paragraph position="3"> Our rst approach is knowledge-free. We put together a fully-connected, uniform trigram source model P(p) over the tokens C, V, and SPACE. Our channel model P(c  |p) is also fully-connected and uniform.</Paragraph>
    <Paragraph position="4"> We allow source as well as channel probabilities to oat during training. This almost works, as shown in Figure 8(b). It correctly clusters letters into vowels and consonants, but assigns exactly the wrong labels! A complex cluster analysis (Finch and Chater, 1991) yields similar results.</Paragraph>
    <Paragraph position="5"> Our second approach uses syllable theory. Our source model generates each source word in three phases. First, we probabilistically select the number of syllables to generate. Second, we probabilistically ll each slot with a syllable type. Every human language has a clear inventory of allowed syllable types, and many languages share the same inventory. Some examplars are (1995):  For our purposes, we allow generation of V, VC, VCC, CV, CVC, CCV, CVCC, CCVC, or CCVCC. Elements of the syllable type sequence are chosen independently of each other, except that we disallow vowel-initial syllables following consonant- nal syllables, following the phonetic universal tendency to maximize the onset (the initial consonant cluster of a syllable). Third, we spell out the chosen syllable types, so that the whole source model yields sequences over the tokens C, V, and SPACE, as before. This spelling-out is deterministic, except that we may turn a V into either one or two Vs, to account for dipthongs. The channel model again maps {C, V} onto {a, b, c, . . . }, and we again run EM to learn both source and channel probabilities.</Paragraph>
    <Paragraph position="6"> Figure 8(c) shows that this almost works. To make it work, 8(d), we force the number of syllables per word in the model to be xed and uniform, rather than learned. This prevents the system from making analyses that are too short. We also execute several EM runs with randomly initialized P(c  |p), and choose the run with the highest resulting P(c).</Paragraph>
    <Paragraph position="7">  (a) primera parte del ingenioso hidalgo don quijote de la mancha (b) VVCVCVC VCVVC VCV CVVCVVCVC VCVCVVC VCV VCVVCVC VC VC VCVVVC (c) CCV.CV.CV CVC.CV CVC VC.CVC.CV.CV CV.CVC.CV CVC CVC.CV.CV CV CV CVC.CCV (d) CCV.CV.CV CVC.CV CVC VC.CV.CV.V.CV CV.CVC.CV CVC CV.V.CV.CV CV CV CVC.CCV (e) NSV.NV.NV NVS.NV NVS VS.NV.SV.V.NV NV.NVS.NV NVS NV.V.NV.NV NV NV NVS.NSV  unsupervised consonant-vowel decipherment, (c) is a decipherment informed by syllable structure, (d) is an improved decipherment, and (e) is a decipherment that also attempts to distinguish sonorous (S) and non-sonorous (N) consonants.</Paragraph>
    <Paragraph position="8"> We see that the Spanish letters are accurately divided into consonants and vowels, and it is also straight-forward to ask about the learned syllable generation probabilities they are CV (0.50), CVC (0.20), V (0.16), VC (0.11), CCV (0.02), CCVC (0.0002).</Paragraph>
    <Paragraph position="9"> As a sanity check, we manually remove all P(c  |p) parameters that match C with Spanish vowel-letters (a, e, i, o, u, y, and accented versions) and V with Spanish consonant-letters (b, c, d, etc), then re-run the same EM learning. We obtain the same P(c).</Paragraph>
    <Paragraph position="10"> Exactly the same method works for Latin. Interestingly, the fully-connected P(c  |p) model leads to a higher P(c) than the correctly constrained channel. We nd that in the former, the letter i is sometimes treated as a vowel and other times as a consonant. The word omnium is analyzed by EM as VC.CV.VC, while iurium is analyzed as CVC.CVC.</Paragraph>
    <Paragraph position="11"> We went a step further to see if EM could identify which letters encode sonorous versus non-sonorous consonants. Sonorous consonants are taken to be perceptually louder, and include n, m, l, and r. Additionally, vowels are more sonorous than consonants. A universal tendency (the sonority hierarchy) is that syllables have a sonority peak in the middle, which falls off to the left and right. This captures why the syllable G R A R G sounds more typical than R G A G R. There are exceptions, but the tendency is strong.</Paragraph>
    <Paragraph position="12"> We modify our source model to generate S (sonorous consonant), N (non-sonorous consonant), V, and SPACE. We do this by changing the spell-out to probabilistically transform CCVC, for example, into either N S V S or N S V N, both of which respect the sonority hierarchy. The result is imperfect, with the EM hijacking the extra symbols. However, if we rst run our C, V, SPACE model and feed the learned model to the S, N, V, SPACE model, then it works fairly well, as shown in Figure 8(e). Learned vowels include (in order of generation probability): e, a, o, u, i, y. Learned sonorous consonants include: n, s, r, l, m. Learned non-sonorous consonants include: d, c, t, l, b, m, p, q. The model bootstrapping is good for dealing with too many parameters; we see a similar approach in Brown et al's (1993) march from Model 1 to Model 5.</Paragraph>
    <Paragraph position="13"> There are many other constraints to explore. For example, physiological constraints make some phonetic combinations more unlikely. AE N T and AE M P work because the second sound leaves the mouth wellprepared to make the third sound, while AE N P does not. These and other constraints complement the model by also working across syllable boundaries. There are also constraints on phoneme inventory (no voiced consonant like B without its unvoiced partner like P) and syllable inventory (no CCV without CV).</Paragraph>
  </Section>
  <Section position="9" start_page="503" end_page="503" type="metho">
    <SectionTitle>
7 Brute-Force Phonetic Decipherment
</SectionTitle>
    <Paragraph position="0"> Another approach to universal phonetic decipherment is to build phoneme n-gram databases for all human languages, then fully decipher with respect to each in turn. At the end, we need an automatic procedure for evaluating which source language has the best t.</Paragraph>
    <Paragraph position="1"> There do not seem to be sizeable phoneme-sequence corpora for many languages. Therefore, we used source character models as a stand in, decoding as in Section 3. We built 80 different source models from sequences we downloaded from the UN Universal Declaration of Human Rights website.1 Suppose our ciphertext starts cevzren cnegr qry...</Paragraph>
    <Paragraph position="2"> as in Figure 9(a). We decipher it against all 80 source language models, and the results are shown in Figure 9(b-f), ordered by post-training P(c). The system believes 9(a) is enciphered Spanish, but if not, then Galician, Portuguese, or Kurdish. Spanish is actually the correct answer, as the ciphertext is again Don Quixote (put through a simple letter substitution to show the problem from the computer's point of view).</Paragraph>
    <Paragraph position="3"> Similarly, EM detects that fpn owoktvcv hu ihgzsnwfv rqcffnw cw... is actually English, and deciphers it as the analysis of wocuments pritten in...</Paragraph>
    <Paragraph position="4"> Many writing systems do not write vowel sounds.</Paragraph>
    <Paragraph position="5"> We can also do a brute force decipherment of vowelless writing by extending our channel model: rst, we deterministically remove vowel sounds (or letters, in the above case), then we probabilistically substitute letters according to P(c  |p). For the ciphertext ceze ceg qy... , EM still proposes Spanish as the best source language, with decipherment prmr prt dl...</Paragraph>
  </Section>
  <Section position="10" start_page="503" end_page="504" type="metho">
    <SectionTitle>
8 Word-Based Decoding
</SectionTitle>
    <Paragraph position="0"> Letter-based substitution/transposition schemes are technically called ciphers, while systems that make whole-word substitutions are called codes. As an example code, one might write I will bring the parrot to  the best decipherments obtained for some of the 80 candidate source languages, automatically sorted by P(c). Canada instead of I will bring the money to John or, one might encode every word in a message. Machine translation has code-like characteristics, and indeed, the initial models of (Brown et al., 1993) took a word-substitution/transposition approach, trained on a parallel text.</Paragraph>
    <Paragraph position="1"> Because parallel text is scarce, it would be very good to extend unsupervised letter-substitution techniques to word-substitution in MT. Success to date has been limited, however. Here we execute a small-scale example, but completely from scratch.</Paragraph>
    <Paragraph position="2"> In this experiment, we know the Arabic cipher names of seven countries: m!lyzy!, !lmksyk, knd!, bryT!ny!, frns!, !str!ly!, and !ndwnysy!. We also know a set of English equivalents, here in no particular order: Mexico, Canada, Malaysia, Britain, Australia, France, and Indonesia. Using non-parallel corpora, can we gure out which word is a translation of which? We use neither spelling information nor exclusivity, since these are not exploitable in the general MT problem.</Paragraph>
    <Paragraph position="3"> To create a ciphertext, we add phrases X Y and Y X to the ciphertext whenever X and Y co-occur in the same sentence in the Arabic corpus. Sorting by frequency, this ciphertext looks like:  etc.</Paragraph>
    <Paragraph position="4"> Each corpus induces a kind of world map, with high frequency indicating closeness. The task is to gure out how elements of the two world maps correspond.</Paragraph>
    <Paragraph position="5"> We train a source English bigram model P(p) on the plaintext, then set up a uniform P(c  |p) channel with 7x7=49 parameters. Our initial result is not good: EM locks up after two iterations, and every English word learns the same distribution. When we choose a random initialization for P(c  |p), we get a better result, as 4 out of 7 English words correctly map to their Arabic equivalents. With 5 random restarts, we achieve 5 correct, and with 40 or more random restarts, all 7 assignments are always correct. (From among the restarts, we select the one with the best post-EM P(c), not the best accuracy on the task.) The learned P(c  |p) dictionary is shown here (correct mappings are marked with *).</Paragraph>
    <Paragraph position="7"/>
  </Section>
class="xml-element"></Paper>