<?xml version="1.0" standalone="yes"?>
<Paper uid="N03-1026">
  <Title>Statistical Sentence Condensation using Ambiguity Packing and Stochastic Disambiguation Methods for Lexical-Functional Grammar</Title>
  <Section position="3" start_page="0" end_page="0" type="metho">
    <SectionTitle>
2 Statistical Sentence Condensation in the
LFG Framework
</SectionTitle>
    <Paragraph position="0"> In this section, each of the system components will be described in more detail.</Paragraph>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
2.1 Parsing and Transfer
</SectionTitle>
      <Paragraph position="0"> In this project, a broad-coverage LFG grammar and parser for English was employed (see Riezler et al. (2002)). The parser produces a set of context-free constituent (c-)structures and associated functional (f-)structures for each input sentence, represented in packed form (see Maxwell and Kaplan (1989)). For sentence condensation we are only interested in the predicate-argument structures encoded in f-structures.</Paragraph>
      <Paragraph position="1"> For example, Fig. 1 shows an f-structure manually selected out of the 40 f-structures for the sentence: A prototype is ready for testing, and Leary hopes to set requirements for a full system by the end of the year.</Paragraph>
      <Paragraph position="2"> The transfer component for the sentence condensation system is based on a component previously used in a machine translation system (see Frank (1999)). It consists of an ordered set of rules that rewrite one f-structure into another. Structures are broken down into flat lists of facts, and rules may add, delete, or change individual facts. Rules may be optional or obligatory. In the case of optional rules, transfer of a single input structure may lead to multiple alternate output structures. The transfer component is designed to operate on packed input from the parser and can also produce packed representations of the condensation alternatives, using methods adapted from parse packing.1 An example rule that (optionally) removes an adjunct is shown below: +adjunct(X,Y), in-set(Z,Y) ?=&gt; delete-node(Z,r1), rule-trace(r1,del(Z,X)).</Paragraph>
      <Paragraph position="3"> This rule eliminates an adjunct,Z, by deleting the fact that Z is contained within the set of adjuncts, Y, associated with the expressionX. The+before theadjunct(X,Y) fact marks this fact as one that needs to be present for the rule to be applied, but which is left unaltered by the rule application. The in-set(Z,Y) fact is deleted. Two new facts are added. delete-node(Z,r1) indicates that the structure rooted at node Z is to be deleted, and rule-trace(r1,del(Z,X)) adds a trace of this rule to an accumulating history of rule applications. This history records the relation of transferred f-structures to the original f-structure and is available for stochastic disambiguation. null  be employed in these experiments since the current interface to the generator and stochastic disambiguation component still requires unpacked representations.</Paragraph>
      <Paragraph position="4"> &amp;quot;A prototype is ready for testing , and Leary hopes to set requirements for a full system by the end of the year.&amp;quot;</Paragraph>
      <Paragraph position="6"> DET[?]FORM the, DET[?]TYPE defDETSPECCASE acc, NUM sg, PERS 3, PFORM of519OBJ ADJUNCT[?]TYPE nominal, PSEM unspecified, PTYPE sem512 ADJUNCT countGRAINNTYPE 'the'PRED DET[?]FORM the, DET[?]TYPE defDETSPECCASE acc, NUM sg, PERS 3, PFORM by469 OBJ ADV[?]TYPE vpadv, PSEM unspecified, PTYPE sem451 ADJUNCT PERF [?]_, PROG [?]_TNS[?]ASPINF[?]FORM to, PASSIVE [?], VTYPE main280 XCOMP MOOD indicative, PERF [?]_, PROG [?]_, TENSE presTNS[?]ASPPASSIVE [?], STMT[?]TYPE decl, VTYPE main252 COORD +_, COORD[?]FORM and, COORD[?]LEVEL ROOT197  slept.), the optional deletion of parts of coordinate structures (e.g., They laughed and giggled. can become They giggled.), and certain simplifications (e.g. It is clear that the earth is round. can become The earth is round. but It seems that he is asleep. cannot become He is asleep.). For example, one possible post-transfer output of the sentence in Fig. 1 is shown in Fig. 2.</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
2.2 Stochastic Selection and Generation
</SectionTitle>
      <Paragraph position="0"> The transfer rules are independent of the grammar and are not constrained to preserve the grammaticality or well-formedness of the reduced f-structures. Some of the reduced structures therefore may not correspond to any English sentence, and these are eliminated from future consideration by using the generator as a filter. The filtering is done by running each transferred structure through the generator to see whether it produces an output string.</Paragraph>
      <Paragraph position="1"> If it does not, the structure is rejected. For example, for the f-structure in Fig. 1, the transfer system proposed 32 possible reductions. After filtering these structures by generation, 16 reduced f-structures comprising possible &amp;quot;A prototype is ready for testing.&amp;quot;</Paragraph>
      <Paragraph position="3"> condensations of the input sentence survive. The 16 well-formed structures correspond to the following strings that were outputted by the generator (note that a single structure may correspond to more than one string and a given string may correspond to more than one structure): A prototype is ready.</Paragraph>
      <Paragraph position="4"> A prototype is ready for testing.</Paragraph>
      <Paragraph position="5"> Leary hopes to set requirements for a full system.</Paragraph>
      <Paragraph position="6"> A prototype is ready and Leary hopes to set requirements for a full system.</Paragraph>
      <Paragraph position="7"> A prototype is ready for testing and Leary hopes to set requirements for a full system.</Paragraph>
      <Paragraph position="8"> Leary hopes to set requirements for a full system by the end of the year.</Paragraph>
      <Paragraph position="9"> A prototype is ready and Leary hopes to set requirements for a full system by the end of the year. A prototype is ready for testing and Leary hopes to set requirements for a full system by the end of the year.</Paragraph>
      <Paragraph position="10"> In order to guarantee non-empty output for the over-all condensation system, the generation component has to be fault-tolerant in cases where the transfer system operates on a fragmentary parse, or produces non-valid f-structures from valid input f-structures. Robustness techniques currently applied to the generator include insertion and deletion of features in order to match invalid transferoutput to the grammar rules and lexicon. Furthermore, repair mechanisms such as repairing subject-verb agreement from the subject's number value are employed. As a last resort, a fall-back mechanism to the original uncondensed f-structure is used. These techniques guarantee that a non-empty set of reduced f-structures yielding grammatical strings in generation is passed on to the next system component. In case of fragmentary input to the transfer component, grammaticaliy of the output is guaranteed for the separate fragments. In other words, strings generated from a reduced fragmentary f-structure will be as grammatical as the string that was fed into the parsing component.</Paragraph>
      <Paragraph position="11"> After filtering by the generator, the remaining f-structures were weighted by the stochastic disambiguation component. Similar to stochastic disambiguation for constraint-based parsing (Johnson et al., 1999; Riezler et al., 2002), an exponential (a.k.a. log-linear or maximumentropy) probability model on transferred structures is estimated from a set of training data. The data for estimation consists of pairs of original sentences y and gold-standard summarized f-structures s which were manually selected from the transfer output for each sentence. For training data {(sj,yj)}mj=1 and a set of possible summarized structures S(y) for each sentence y, the objective was to maximize a discriminative criterion, namely the conditional likelihood L(l) of a summarized f-structure given the sentence. Optimization of the function shown below was performed using a conjugate gradient optimization routine:</Paragraph>
      <Paragraph position="13"> At the core of the exponential probability model is a vector of property-functions f to be weighted by parameters l. For the application of sentence condensation, 13,000 property-functions of roughly three categories were used:  arrive at the reduced f- structures ([?] 60 properties). A trained probability model is applied to unseen data by selecting the most probable transferred f-structure, yielding the string generated from the selected structure as the target condensation. The transfered f-structure chosen for our current example is shown in Fig. 3.</Paragraph>
      <Paragraph position="14">  This structure was produced by the following set of transfer rules, where var refers to the indices in the representation of the f-structure:</Paragraph>
      <Paragraph position="16"> These rules delete the adjunct of the first conjunct (for testing), the adjunct of the second conjunct (by the end of the year), the rest of the second conjunct (Leary hopes to set requirements for a full system), and the conjunction itself (and).</Paragraph>
    </Section>
  </Section>
  <Section position="4" start_page="0" end_page="0" type="metho">
    <SectionTitle>
3 A Method for Automatic Evaluation of
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
Sentence Summarization
Evaluation of quality of sentence condensation systems,
</SectionTitle>
      <Paragraph position="0"> and of text summarization and simplification systems in general, has mostly been conducted as intrinsic evaluation by human experts. Recently, Papineni et al.'s (2001) proposal for an automatic evaluation of translation systems by measuring n-gram matches of the system output against reference examples has become popular for evaluation of summarization systems. In addition, an automatic evaluation method based on context-free deletion decisions has been proposed by Jing (2000). However, for summarization systems that employ a linguistic parser as an integral system component, it is possible to employ the standard evaluation techniques for parsing directly to an evaluation of summarization quality. A parsing-based evaluation allows us to measure the semantic aspects of summarization quality in terms of grammaticalfunctional information provided by deep parsers. Furthermore, human expertise was necessary only for the creation of condensed versions of sentences, and for the manual disambiguation of parses assigned to those sentences. Given such a gold standard, summarization quality of a system can be evaluated automatically and repeatedly by matching the structures of the system output against the gold standard structures. The standard metrics of precision, recall, and F-score from statistical parsing can be used as evaluation metrics for measuring matching quality: Precision measures the number of matching structural items in the parses of the system output and the gold standard, out of all structural items in the system output's parse; recall measures the number of matches, out of all items in the gold standard's parse. F-score balances precision and recall as (2 x precision x recall)/(precision + recall).</Paragraph>
      <Paragraph position="1"> For the sentence condensation system presented above, the structural items to be matched consist of relation(predicate, argument) triples. For example, the gold-standard f-structure of Fig. 2 corresponds to 23 dependency relations, the first 14 of which are shared with the reduced f-structure chosen by the stochastic disambigua- null Matching these f-structures against each other corresponds to a precision of 1, recall of .61, and F-score of .76.</Paragraph>
      <Paragraph position="2"> The fact that our method does not rely on a comparison of the characteristics of surface strings is a clear advantage. Such comparisons are bad at handling examples which are similar in meaning but differ in word order or vary structurally, such as in passivization or nominalization. Our method handles such examples straightforwardly. Fig. 4 shows two serialization variants of the condensed sentence of Fig. 2. The f-structures for these examples are similar to the f-structure assigned to the gold standard condensation shown in Fig. 2 (except for the relations ADJUNT-TYPE:parenthetical versus ADV-TYPE:vpadv versus ADV-TYPE:sadv). An evaluation of summarization quality that is based on matching f-structures will treat these examples equally, whereas an evaluation based on string matching will yield different quality scores for different serializations.</Paragraph>
      <Paragraph position="3"> &amp;quot;A prototype, for testing, is ready.&amp;quot;  In the next section, we present experimental results of an automatic evaluation of the sentence condensation system described above. These results show a close correspondence between automatically produced evaluation results and human judgments on the quality of generated condensed strings.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>