<?xml version="1.0" standalone="yes"?>
<Paper uid="H94-1053">
  <Title>STATISTICAL LANGUAGE PROCESSING USING HIDDEN UNDERSTANDING MODELS</Title>
  <Section position="4" start_page="0" end_page="278" type="metho">
    <SectionTitle>
-2 EXPRESSING MEANINGS
</SectionTitle>
    <Paragraph position="0"> One of the key requirements for a hidden understanding model is that the meaning representation must be both expressive and appropriate for automatic learning techniques. Logical notations, such as the predicate calculus, are generally considered to possess sufficient expressive power. The difficulty lies in finding a meaning representation that can be  readily aligned to the words of a sentence, and for which there is a tractable probability model for meanings. To satisfy these requirements, we have developed a family of representations which we call tree structured meaning representations.</Paragraph>
  </Section>
  <Section position="5" start_page="278" end_page="278" type="metho">
    <SectionTitle>
2.1 TREE STRUCTURED MEANING
REPRESENTATIONS
</SectionTitle>
    <Paragraph position="0"> The central characteristic of a tree structured representation is that individual concepts appear as nodes in a tree, with component concepts appearing as nodes attached directly below them. For example, the concept of aflight in the ATIS domain has component concepts including airline,flight number, origin, and destination. These could then form part of the representation for the phrase: United flight 203 from Dallas to Atlanta. We require that the order of the component concepts must match the order of the words they correspond to. Thus, the representation of the phrase flight 203 to Atlanta from Dallas on United includes the same nodes as the earlier example, but in a different order. For both examples, the interpretation is identical. More formally, the meaning of a tree structured representation is invariant with respect to the left-to-fight order of the component concept nodes.</Paragraph>
    <Paragraph position="1"> At the leaves of a meaning tree are the words of the sentence.</Paragraph>
    <Paragraph position="2"> We distinguish between nodes that appear above other nodes, and those that appear directly above the words. These will be referred to as nonterminal nodes and terminal nodes respectively, forming two disjoint sets. No node has both words and other nodes appearing directly below it. In the current example, aflight node represents the abstract concept of a flight, which is a structured entity that may contain an origin, a destination, and other component concepts. Appearing directly above the word &amp;quot;flight&amp;quot; is a terminal node, which we call a fight indicator. This name is chosen to distinguish it from the flight node, and also because the word &amp;quot;flight,&amp;quot; in some sense, indicates the presence of a flight concept. Similarly, there are airline indicators, origin indicators, and destination indicators* These nodes can be thought of as elements in a specialized sublanguage for expressing meaning in the ATIS domain.</Paragraph>
  </Section>
  <Section position="6" start_page="278" end_page="280" type="metho">
    <SectionTitle>
3 THE STATISTICAL MODEL
</SectionTitle>
    <Paragraph position="0"> One central characteristic of hidden understanding models is that they are generative. From this viewpoint, language is produced by a two component statistical process. The first component chooses the meaning to be expressed, effectively deciding &amp;quot;what to say&amp;quot;. The second component selects word sequences to express that meaning, effectively deciding &amp;quot;how to say it&amp;quot;. The first phase is referred to as the semantic language model, and can be thought of as a stochastic process that produces meaning expressions selected from a universe of meanings. The second phase is referred to as the lexical realization model, and can be thought of as a stochastic process that generates words once a meaning is given.</Paragraph>
    <Paragraph position="1">  By analogy with hidden Markov models, we refer to the combination of these two models as a hidden understanding model. The word hidden refers to the fact that only words can be observed. The internal states of each of the two models are unseen and must be inferred from the words. The problem of language understanding, then, is to recover the most likely meaning structure given a sequence of words. More formally, understanding a word sequence W is accomplished by searching among all possible meanings for some meaning M such that P(MIW) is maximized. By Bayes Rule, P(MIW) can be rewritten as: ) nonterminal nodes</Paragraph>
    <Paragraph position="3"> Now, since P(W) does not depend on M, maximizing P(MIW) is equivalent to maximizing the product P(W1M) P( M).</Paragraph>
    <Paragraph position="4"> However, P(W1M) is simply our !exical realization model, and P(M) is simply our semantic language model. Thus, by searching a combination of these models it is possible to fred the most likely meaning M given word sequence W.</Paragraph>
    <Section position="1" start_page="279" end_page="279" type="sub_section">
      <SectionTitle>
3.1 Semantic Language Model
</SectionTitle>
      <Paragraph position="0"> For tree structured meaning representations, individual nontenninal nodes determine particular abstract semantic concepts. In the semantic language model, each abstract concept corresponds to a probabilistic state transition network. All such networks are then combined into a single probabilistic recursive transition network, forming the entire semantic language model.</Paragraph>
      <Paragraph position="1"> The network corresponding to a particular abstract concept consists of states for each of its component concepts, together with two extra states that define the entry and exit points. Every component concept is fully connected to every other component concept, with additional paths leading from the entry state to each component concept, and from each component concept to the exit state. Figure 4 shows a sample network corresponding to the flight concept. Of course, there are many more flight component concepts in the ATIS domain than actually appear in this example.</Paragraph>
      <Paragraph position="2"> Associated with each arc is a probability value, in a similar fashion to the TINA system \[Seneff, 92\]. These probabilities have the form P(State,,IStaten_l,Context), which is the probability of taking a transition from one state to another within a particular context. Thus, the arc from origin to dest has probability P(dest I origin,flight), meaning the probability of entering dest from origin within the context of the flight network. Presumably, this probability is relatively high, since people usually mention the destination of a flight directly after mentioning its origin. Conversely, P(origin I dest,flight) is probably low because people don't usually express concepts in that order. Thus, while all paths through the state space are possible, some have much higher probabilities than others.</Paragraph>
      <Paragraph position="3"> Within a concept network, component concept states exist for both nonterminal concepts, such as origin, as well as terminal concepts, such as flight indicator. Arrows pointing into nonterminal states indicate entries into other networks, while arrows pointing away indicate exits out of those networks.</Paragraph>
      <Paragraph position="4"> Terminal states correspond to networks as well, although these are determined by the lexical realization model and have a different internal structure. Thus, every meaning tree directly corresponds directly to some particular path through the state space. Figure 5 shows a meaning tree and its corresponding path through the state space.</Paragraph>
    </Section>
    <Section position="2" start_page="279" end_page="280" type="sub_section">
      <SectionTitle>
3.2 Lexical Realization Model
</SectionTitle>
      <Paragraph position="0"> Just as nonterminal tree nodes correspond to networks in the semantic language model, terminal nodes correspond to networks in the lexical realization model. The difference is that semantic language networks specify transition probabilities between states, while lexical realization networks specify transition probabilities between words. Lexical realization  probabilities have the form P( word n l wordn_ l ,context ) , which is the probability of taking a transition from one word to another given a particular context. Thus, P(showlplease, show-indicator ) is the probability that the word show follows the word please within the context of a show indicator phrase. In addition, there are two pseudo-words, *begin* and *end*, which indicate the beginning and ending of phrases. Thus, we have probabilities such as P(please\[*begin*,show-indicator ), which is the probability that please is the first word of a show indicator phrase, and P(*end*lme, show-indicator ), which is the probability of exiting a show indicator phrase given that the previous word was me.</Paragraph>
    </Section>
  </Section>
  <Section position="7" start_page="280" end_page="280" type="metho">
    <SectionTitle>
4 THE UNDERSTANDING COMPONENT
</SectionTitle>
    <Paragraph position="0"> As we have seen, understanding a word string W requires finding a meaning M such that the probability P(W\]M) P(M) is maximized. Since, the semantic language model and the lexical realization model are both probabilistic networks, P(WJM) P(M) is the probability of a particular path through the combined network. Thus, the problem of understanding is to fred the highest probability path among all possible paths, where the probability of a path is the product of all the transition probabilities along that path.</Paragraph>
    <Paragraph position="1"> rr \[ P( state.lstaten_ l,context) if t in Semantic Language Model\] P(rat~)=,l~hLP(word, lword,_t,context) .... if t in Lexieal Realization ModelJ / Thus far, we have discussed the need to search among all meanings for one with a maximal probability. In fact, if it were necessary to search every path through the combined network individually, the algorithm would require exponential time with respect to sentence length. Fortunately, this can be drastically reduced by combining the probability computation of common subpaths through dynamic programming. In particular, because our meaning representation aligns to the words, the search can be efficiently performed using the well-known Viterbi \[Viterbi, 67\] algorithm.</Paragraph>
    <Paragraph position="2"> Since our underlying model is a recursive transition network, the states for the Viterbi search must be allocated dynamically as the search proceeds. In addition, it is necessary to prune very low probability paths in order to keep the computation tractable. We have developed an elegant algorithm that integrates state allocation, Viterbi search, and pruning all within a single traversal of a tree-like data structure.</Paragraph>
  </Section>
  <Section position="8" start_page="280" end_page="281" type="metho">
    <SectionTitle>
5 THE TRAINING COMPONENT
</SectionTitle>
    <Paragraph position="0"> In order to train the statistical model, we must estimate transition probabilities for the semantic language model and lexical realization model. In the ease of fully specified meaning trees, each meaning tree can be straightforwardly converted into a path through state space. Then, by counting occurrence and transition frequencies along those paths, it is possible to fonn simple estimates of the transition probabilities. Let C(statem,contexts) denote the number of times state m has occurred in contexts, and let C(statenlstatem,contexts)denote the number of times that this condition has led to a transition to state state n. Similarly, define counts C(wordm,contextt) and C(wordnlwordm,contextt). Then, a direct estimate of the probabilities is given by: and Jb (state n lstate re,context) C( statenlstatem ,cdegntext ) , C ( state m , context ) A C(word word context) P(wordnlwordm,context) = n m' .</Paragraph>
    <Paragraph position="1"> C( word m , context ) show flight &amp;quot; ~ .</Paragraph>
    <Paragraph position="2"> Show flights to Atlanta  In order to obtain robust estimates, these simple estimates are smoothed with backed-off estimates \[Good, 53\], using techniques similar to those used in speech recognition \[Katz, 87; Placeway et al., 93\]. Thus, P(statenlstatem,context ) is smoothed with 1~( statenJ,context ), and P( wordnJ word re,context ) is smoothed with 15(wordnlcontext). Robustness is further increased through word classes. For example, Boston and San Francisco are both members of the class of cities.</Paragraph>
  </Section>
class="xml-element"></Paper>