<?xml version="1.0" standalone="yes"?>
<Paper uid="P98-2147">
  <Title>mar, A language-Independent System for Parsing Unrestricted Tezt. Mouton de</Title>
  <Section position="3" start_page="891" end_page="895" type="intro">
    <SectionTitle>
2. Algorithm
</SectionTitle>
    <Paragraph position="0"> We will start by giving a precise definition of dynamic grammars. We will then explain each stage of grammar compilation. Grammar compilation takes as input a weighted CFG represented as a weighted transducer (Salomaa and Soittola, 1978), which may have been optimized prior to compilation (preoptimized). The weighted transducer is analyzed by the compilation algorithm, and the analysis, if successful, outputs a collection of weighted automata that are combined at runtime according to the current dynamic grammar configuration and the strings being recognized. Since not all CFGs can be compiled into weighted automata, the compilation algorithm may reject an input grammar. The class of allowed grammars will be defined later.</Paragraph>
    <Section position="1" start_page="891" end_page="892" type="sub_section">
      <SectionTitle>
2.1. Dynamic grammars
</SectionTitle>
      <Paragraph position="0"> The following notation will be used in the rest of the paper. A weighted CFG G = (V,P) over the alphabet E, with real-number weights consists of a finite alphabet V of variables or nonterminals disjoint from ~, and a finite set</Paragraph>
      <Paragraph position="2"> tion rules (Autebert et al., 1997). Given strings u, v E (V U ~)*, and real numbers c and c', we write (u, c) 2+ (v, c') when there is a derivation from u with weight c to v with weight c'. We denote by La(X) the weighted language generated by a nonterminal X: LG(X) = {(w,c) E ~* x R: (X, 0) -~ (w,c)} We can now define the two grammar-changing operations that we use.</Paragraph>
      <Paragraph position="3"> Dynamic activation or deactivation of rules 2 We augment the grammar with a set of active nonterminals, which are those available as start symbols for derivations. More precisely, let A C_ V be the set of active nonterminals. The language generated by G is then</Paragraph>
      <Paragraph position="5"> minals, and the rules involving them, are available for use in derivations; they are just not available as start symbols. Dynamic rule activation or deactivation is just the dynamic redefinition of the set A in successive uses of the grammar.</Paragraph>
      <Paragraph position="6"> Dynamic substitution Let a be a weighted rational transduction of ~* to A* x R, ~ C_ A, that is a regular weighted substitution (Berstel, 1979). a is a monoid morphism verifying: 2This is the terminology used in this area, though a more appropriate expression would be dynamic activation or deactivation of nonterminal symbols.  Vx E ~, a(x) C Reg(A&amp;quot; x R) where Reg(A* x R) denotes the set of weighted regular languages over the alphabet A. Thus a simply substitutes for each symbol a E ~ a weighted regular expression a(a). A dynamic substitution consists of the application of the substitution a to ~, during the process of recognition of a word sequence. Thus, after substitution, the language generated by the new</Paragraph>
      <Paragraph position="8"> Our algorithm allows for both of those dynamic grammar changes without recompiling the grammar.</Paragraph>
    </Section>
    <Section position="2" start_page="892" end_page="892" type="sub_section">
      <SectionTitle>
2.2. Preprocessing
</SectionTitle>
      <Paragraph position="0"> Our compilation algorithm operates on a weighted transducer v(G) encoding a factored representation of the weighted CFG G, which is generated from G by a separate preprocessor. This preprocessor is not strictly needed, since we could use a version of the main algorithm that works directly on G. However, pre-processing can improve dramatically the time and space needed for the main compilation step, since the preprocessor uses determinization and minimization algorithms for weighted transducers (Mohri, 1997) to increase the sharing -- factoring- among grammar rules that start or end the same way.</Paragraph>
      <Paragraph position="1"> The preprocessing step builds a weighted transducer in which each path corresponds to a grammar rule. Rule X(~ -+ Y1 --.Y~ has a corresponding path that maps X to the sequence I/1 ...Y~ with weight ~. For example, the small CFG in Figure 1 is preprocessed into the compacted transducer shown in Figure 2.</Paragraph>
    </Section>
    <Section position="3" start_page="892" end_page="894" type="sub_section">
      <SectionTitle>
2.3. Compilation
</SectionTitle>
      <Paragraph position="0"> The compilation of weighted left-linear or right-linear grammars into weighted automata is straightforward (Aho and Ullman, 1973). In the right-linear case, for instance, the states of the automaton are the grammar nonterminals together with a new final state F. There is a</Paragraph>
      <Paragraph position="2"> transition labeled with a E E and weight a E R from X E V to Y E V iff the grammar contains the rule Xa --+ aY. There is a transition from X to F labeled with a and weight a iff Xa --~ a is a rule of the grammar. The initial states are the states corresponding to the active nonterminals. For example, Figure 3 shows the weighted automaton for grammar G2 consisting of the last three rules of G1 with start symbol X.</Paragraph>
      <Paragraph position="3"> However, the standard methods for left- and right-linear grammars cannot be used for grammars such as G1 that generate regular sets but have rules that are neither left- nor right-linear. But we can use the methods for left- and right-linear grammars as subroutines if the grammar can be decomposed into left-linear and right-linear components that do not call each other recursively (Pereira and Wright, 1997). More precisely, define a dependency graph Dc for G's nonterminals and examine the set of its strongly-connected components (SCCs). 4 The nodes of Da are G's nonterminals, and there is a directed edge from X to Y if Y appears in the right-hand side of a rule with left-hand side X, that is, if the definition of X depends  on Y. Each SCC S of DG has a corresponding subgrammar of G consisting of those rules with 4 Recall that the strongly connected components of a directed graph are the equivalence classes of graph nodes under the relation R defined by: q R q~ if q~ can be reached from q and q from q~.</Paragraph>
      <Paragraph position="4">  G1.</Paragraph>
      <Paragraph position="5"> left-hand nonterminals in S, with nonterminals not in S treated as terminal symbols. If each of these subgrammars is either left-linear or rightlinear, we shall see that compilation into a single finite automaton is possible.</Paragraph>
      <Paragraph position="6"> The dependency graph DG can be obtained easily from the transducer r(G). For example, Figure 4 shows the dependency graph for our example grammar G1, with SCCs {Z} and (X, Y}. It is clear that G1 satisfies our condition, and Figure 5 shows the result of compiling</Paragraph>
      <Paragraph position="8"> The SCCs of Da can be obtained in time linear in the size of G (Aho et hi., 1974). Before starting the compilation, we check that each subgrammar is left-linear or right-linear (as noted above, nonterminals not in the SCC of a subgrammar are treated as terminals). For example, if (X1, X2} is an SCC, then the sub-</Paragraph>
      <Paragraph position="10"> with X1,X2, Y1,Y2 E V and a,b E ~ is rightlinear, since expressions such as aYlbY2 can be treated as elements of the terminal alphabet of the subgrammar.</Paragraph>
      <Paragraph position="11"> When the compilation condition holds, for each SCC S we can build a weighted automaton K(S) representing the language of S's sub-grammar using the standard methods. Since some nonterminals of G are treated as terminal symbols within a subgrammar, the transitions of an automaton K(S) may be labeled with nonterminals not in S. 5 The nonterminals not in S can then be replaced by their corresponding automata. The replacement operation is lazy, that is, the states and transitions of the replacing automata are only expanded when needed for a given input string. Another interesting characteristic of our algorithm is that the weighted automata K(S) can be made smaller by determinization and minimization, leading to improvements in runtime performance.</Paragraph>
      <Paragraph position="12"> The automaton M(X) that represents the language generated by nonterminal symbol X can be defined using K(S), where S is the strongly connected component containing X, X E S. For instance, when the subgrammar of S is right-linear, M(X)is the automaton that has the same states, transitions, and final states as K(S) and has the state corresponding to X as initial state. For example, Figure 6 shows K((X,Y)) for G1. M(X) is then obtained from K((X,Y}) by taking X as initial state. The left-linear case can be treated in a similar way. Thus, M(X) can always be defined in constant time and space by editing the automaton K(S). We use a lazy implementation of this editing operation for the definition 5More precisely, they can only be part of other strongly connected components that come before S in a reverse topological sort of the components. This guarantees the convergence of the replacement of the nonterminals by the corresponding automata.</Paragraph>
      <Paragraph position="13">  terminals: A = {X, Y, Z}.</Paragraph>
      <Paragraph position="14"> of the automata M(X): the states and transitions of M(X) are determined using K(S) only when necessary for the given input string. This allows us to save both space and time by avoiding a copy of K(S) for each X E S.</Paragraph>
      <Paragraph position="15"> Once the automaton representing the language generated by each nonterminal is created, we can define the language generated by G by building an automaton Ma with one initial state and one final state, and transitions labeled with active nonterminals from the initial to the final state. Figure 7 illustrates this in the case where A -- {X, Y, Z}.</Paragraph>
      <Paragraph position="16"> Given this construction, the dynamic activation or deactivation of nonterminals can be done by modifying the automaton MG. This operation does not require any recompilation, since it does not affect the automaton M(X) built for each nonterminal X.</Paragraph>
      <Paragraph position="17"> All the steps in building the automata M(X) -- construction of DG, finding the SCCs, and computing for K(S) for each SCC S -- require linear time and space with respect to the size of G. In fact, since we first convert G into a compact weighted transducer r(G), the total work required is linear in the size of r(G). 6 This leads to significant gains as shown by our experiments.</Paragraph>
      <Paragraph position="18"> In summary, the compilation algorithm has the following steps:  1. Build the dependency graph Da of the grammar G.</Paragraph>
      <Paragraph position="19"> 2. Compute the SCCs of Da. 7 3. For each SCC S, construct the automaton  K(S). For each X E S, build M(X) from SApplying the algorithm to a compacted weighted transducer r(G) involves various subtleties that we omit for simplicity.</Paragraph>
      <Paragraph position="20"> TWe order the SCCs in reverse topological order, but this is not necessary for the correctness of the algorithm. K(X). 8 4. Create a simple automaton MG accepting exactly the set of active nonterminals A.</Paragraph>
      <Paragraph position="21"> 5. The automaton is then expanded on-the-fly for each input string using lazy replacement and editing.</Paragraph>
      <Paragraph position="22"> The dynamic substitution of a terminal symbol a by a weighted automaton 9 aa is done by replacing the symbol a by the automaton aa, using the replacement operation discussed earlier. This replacement is also done on demand, with only the necessary part of aa being expanded for a given input string. In practice, the automaton aa can be large, a list of city or person names for example. Thus a lazy implementation is crucial for dynamic substitutions.</Paragraph>
      <Paragraph position="23">  3. Optimizations, Experiments and</Paragraph>
    </Section>
    <Section position="4" start_page="894" end_page="895" type="sub_section">
      <SectionTitle>
Results
</SectionTitle>
      <Paragraph position="0"> We have a full implementation of the compilation algorithm presented in the previous section, including the lazy representations that are crucial in reducing the space requirements of speech recognition applications. Our implementation of the compilation algorithm is part of a genera\] set of grammar tools, the GRM Library (Mohri, 1998b), currently used in speech processing projects at AT&amp;T Labs. The GRM Library also includes an efficient compilation too\] for weighted context-dependent rewrite rules (Mohri and Sproat, 1996) that is used in text-to-speech projects at Lucent Bell Laboratories.</Paragraph>
      <Paragraph position="1"> Since the GRM library is compatible with the FSM general-purpose finite-state machine library (Mohri et al., 1998a), we were able to use the tools provided in FSM library to optimize the input weighted transducers r(G) and the weighted automata in the compilation output.</Paragraph>
      <Paragraph position="2"> We did several experiments that show the efficiency of our compilation method. A key feature of our grammar compilation method is the representation of the grammar by a weighted transducer that can then be preoptimized using weighted transducer determinization and minimization (Mohri, 1997; Mohri, 1998a). To show SFor any X, this is a constant time operation. For instance, if K(S) is right-llnear, we just need to pick out the state associated to X in K(X).</Paragraph>
      <Paragraph position="3"> 9In fact, our implementation allows more generally dynamic substitutions by weighted transducers.</Paragraph>
      <Paragraph position="5"> the benefits of this representation, we compared the compilation time and the size of the resulting lazy automata with and without preoptimization. The advantage of preoptimization would be even greater if the compilation output were fully expanded rather than on-demand.</Paragraph>
      <Paragraph position="6"> We did experiments with full bigram models with various vocabulary sizes, and with two unweighted grammars derived by feature instantiation from hand-built feature-based grammars (Pereira and Wright, 1997). Figure 8 shows the compilation times of full bigram models with and without preoptimization, demonstrating the importance of the optimization allowed by using a transducer representation of the grammar. For a 250-word vocabulary model, the compilation time is about 50 times faster with the preoptimized representation. 1deg Figure 8 also shows the sizes of the resulting lazy automata in the two cases. While in the preoptimized case time and s_~ace grow linearly with vocabulary size (O(x/IGI)), they grow quadratically in the unoptimized case (O(\[G\[)).</Paragraph>
      <Paragraph position="7"> The bigram examples also show the advantages of lazy replacement and editing over the full expansion used in previous work (Pereira and Wright, 1997). Indeed, the size of the fully-expanded automaton for the preoptimized  case grows quadratically with the vocabulary size (O(IGI)), while it grows with the cube of the vocabulary size in the unoptimized case (0(IGt3/2)). For example, compilation is about 700 times faster in the optimized case for a fully expanded automaton even for a 40-word vocabulary model, and the result about 39 times smaller.</Paragraph>
      <Paragraph position="8"> Our experiments with a small and a medium-sized CFG obtained from feature-based grammars confirm these observations (Table 1).</Paragraph>
      <Paragraph position="9"> If dynamic grammars and lazy expansion are not needed, we can expand the result fully and then apply weighted determinization and minimization algorithms. Additional experiments show that this can yield dramatic reductions in automata size.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>