File Information
File: 05-lr/acl_arc_1_sum/cleansed_text/xml_by_section/metho/95/m95-1006_metho.xml
Size: 30,448 bytes
Last Modified: 2025-10-06 14:14:00
<?xml version="1.0" standalone="yes"?> <Paper uid="M95-1006"> <Title>Regarding ST</Title> <Section position="4" start_page="55" end_page="55" type="metho"> <SectionTitle> KEY SYSTEM FEATURE S </SectionTitle> <Paragraph position="0"> Two key design features of PLUM are : statistical language modeling with the associated learning algorithms, an d partial understanding . The first key feature is the use of statistical modeling to guide processing . For the version of PLUM used in MUC-6, part of speech information was determined by using well-known Markov modelin g techniques embodied in BBN's part-of-speech tagger POST (Weischedel, et al ., 1993).</Paragraph> <Paragraph position="1"> A second key feature is partial understanding, by which we mean that all components of PLUM are designed t o operate on partially interpretable input, taking advantage of information when available and not failing when it i s unavailable. Neither a complete grammatical analysis nor complete semantic interpretation is required. The system pieces together a model of the whole from the parts of the text it can understand .</Paragraph> </Section> <Section position="5" start_page="55" end_page="61" type="metho"> <SectionTitle> PLUM PROCESSING STAGE S </SectionTitle> <Paragraph position="0"> The PLUM architecture is presented in Figures 1 and 2 . Ovals represent declarative knowledge bases; rectangles represent processing modules . A more detailed description of the system components, their individual outputs, an d their knowledge bases is presented in Ayuso et al ., 1993. The processing modules are briefly described below.</Paragraph> <Paragraph position="1"> The three MUC-6 systems represent three different levels of complexity . The more complex systems are built on top of the simpler systems in order to minimize duplication of effort and maximize knowledge transfer . The NE task is the simplest task and makes use of only lightweight processes, the first three modules of the PLUM system (th e message reader, the morphological analyzer, and the lexical pattern matcher) .</Paragraph> <Paragraph position="2"> The TE task takes the entity names found by the NE system, and merges multiple references to the same entit y using syntactic and semantic information . The knowledge bases of TE are inherited by ST and do not includ e domain-specific knowledge . Domain-specific knowledge is localized only in ST .</Paragraph> <Paragraph position="3"> The NE system is written completely in C and can either be run as a standalone system or as a server which can be queried by the TE and ST systems, which are written in Lisp .</Paragraph> <Section position="1" start_page="55" end_page="55" type="sub_section"> <SectionTitle> Message Reader </SectionTitle> <Paragraph position="0"> The input to the PLUM system is a file containing one or more messages. The message reader module determines message boundaries, identifies the message header information, and determines paragraph and sentence boundaries .</Paragraph> <Paragraph position="1"> The standalone NE system uses a different message reader than the TE and ST systems .</Paragraph> <Paragraph position="2"> N E The NE system uses a generic SGML parser to read messages . A parameter file lists the SGML tags relevant t o the task, in this case <HL>, <TXT>, <DATELINE>, and <DD> . All other SGML tag pairs are read but ignored. TE/ST The TE/ST system uses a more complex message reader . The specification of the input format is declarative , allowing the system to be easily adapted to handle different message formats . This more sophisticated reader can not only pass the portions of the message on to the rest of the system for processing, but can also extract header information (e.g. the document number) from the message and save that information to become part of a template.</Paragraph> </Section> <Section position="2" start_page="55" end_page="55" type="sub_section"> <SectionTitle> Morphological Analyzer </SectionTitle> <Paragraph position="0"> The first phase of processing in both the C and Lisp systems is assignment of part-of-speech information, e .g. , proper noun, verb, adjective, etc . In BBN' s part-of-speech tagger POST [5], a bi-gram probability model, frequency models for known words (derived from large corpora), and probabilities based on word endings for unknown words ar e employed to assign part of speech to the highly ambiguous words and unknown words of the corpus . POST tags each word with one of 47 possible tags with 97% accuracy for known words . Below are the part-of-speech tag s produced by POST for the following sentence from the walkthrough message : &quot;And concentrate on his duties as rear commodore at the New York Yacht Club .&quot; ((And CC) (concentrate VB) (on IN) (his PP$) (duties NNS) (as IN) (rear JJ) (commodore NN) (at IN ) (the DT) (New York NP) (Yacht NP) (Club NP) ( . .))</Paragraph> </Section> <Section position="3" start_page="55" end_page="57" type="sub_section"> <SectionTitle> Lexical Pattern Matcher </SectionTitle> <Paragraph position="0"> The Lexical Pattern Matcher was developed after MUC-4 to deal with grammatical forms, such as corporatio n names. It applies finite state patterns to the input, which consists of word tokens with part-of-speech and semantic concept information . In particular, word groups that are important to the domain and that may be detectable wit h only local syntactic analysis can be treated here . When a pattern is matched, a semantic form is assigned by the pattern. In the NE system, patterns were used to recognize all three of the expression types which make up the tas k (entity expressions, temporal expressions, and numerical expressions) . The TE and ST systems gather the results of the NE system's processing and incorporate them in the form of lexicon additions . The TE and ST systems contai n no lexical patterns of their own, relying entirely on the domain-independent patterns within the NE system .</Paragraph> <Paragraph position="1"> Continuing with the example sentence discussed above, a pattern recognized the sequence (New York NP) (Yach t NP) (Club NP) as an organization ; the pattern's action substituted the single token (New York Yacht Club NP) wit h semantics of organization . The Lexical Pattern Matcher is the final step in the processing done by the NE system . The set of recognized entities is used by the output functions to SGML-mark the input .</Paragraph> </Section> <Section position="4" start_page="57" end_page="57" type="sub_section"> <SectionTitle> Fast Partial Parser (FPP) </SectionTitle> <Paragraph position="0"> The FPP is a near-deterministic parser which generates one or more non-overlapping parse fragments spanning th e input sentence, deferring any difficult decisions on attachment ambiguities . When cases of permanent, predictable ambiguity arise, the parser finishes the analysis of the current phrase and begins the analysis of a new phrase .</Paragraph> <Paragraph position="1"> Theref/ re, the entities mentioned and some relations between them are f rocessed in every sentence, whethe r syntactically ill-formed, complex, novel, or straightforward . Furthermore, this parsing is done using essentially domain-independent syntactic information .</Paragraph> <Paragraph position="2"> FPP averages about 10 fragments for sentences as complex as in the ST corpus ; this number is inflated since punctuation usually results in an isolated fragment . Figure 3 shows parse fragments for two sentences whic h generated the bulk of the succession information in the walkthrough message .</Paragraph> </Section> <Section position="5" start_page="57" end_page="61" type="sub_section"> <SectionTitle> Semantic Interpreter </SectionTitle> <Paragraph position="0"> The semantic interpreter contains two sub-components : a rule-based fragment interpreter and a pattern-base d sentence interpreter. The first has been used since MUC-3 . The rule-based fragment interpreter applies semanti c rules to each fragment produced by FPP in a bottom-up, compositional fashion . Semantic rules are matched based on general syntactic patterns, using wildcards and similar mechanisms to provide robustness . A semantic rule creates a semantic representation of the phrase as an annotation on the syntactic parse . A semantic formula includes a variable (e.g., ?13), its type and a collection of predicates on that variable . There are three basic types of semanti c forms: entities, events, and states of affairs. Each of these can be further categorized as known, unknown, and referential . Entities correspond to the people, places, things, and time intervals of the domain . These are related in various ways, such as through events (who did what to whom) and states of affairs (properties of the entities) .</Paragraph> <Paragraph position="1"> Entity descriptions typically arise from noun phrases ; events and states of affairs are often described in clauses .</Paragraph> <Paragraph position="2"> The rule-based fragment interpreter encodes defaults so that missing semantic information does not produce errors , but marks elements or relationships as unknown. Partial understanding is critical to text processing systems, a s missing data is normal . For example, the generic predicate PP-MODIFIER indicates that two entities are connecte d via a certain preposition. In this way, the system has a &quot;placeholder&quot; for the information that a certain structura l relation holds, even though it does not know what the semantic relation is . Sometimes understanding the relatio n more fully is of no consequence, since the information does not contribute to the template-filling task. The information is maintained, however, so that later expectation-driven processing can use it if necessary .</Paragraph> <Paragraph position="3"> Due to the fragmentation produced by FPP, top-level constituents are typically more shallow and less varied tha n full sentence parses . A fairly high level of semantics coverage can be obtained quite quickly when the system i s moved to a new domain . This would not be possible if the semantic rules were required to cover a wider variety o f syntactic structures before it could achieve reasonable performance . In this way, semantic coverage can be adde d gradually, while the rest of the system is progressing in parallel .</Paragraph> <Paragraph position="4"> The second sub-component of the semantic interpreter module is a pattern-based sentence interpreter which applie s semantic pattern-action rules to the semantics of each fragment of the sentence . The semantic pattern matching 'omponent employs the same core engine as the lexical pattern matcher . These semantic rules can add additional long-distance relations between semantic entities in different fragments within a sentence . A typical TE-level pattern would seek to attach descriptions to organizations, while a ST-level rule would find a potential SUCCESSION an d attach the PERSON, ORGANIZATION and POST information related to it. Below is an example of a sentence-level rule which looks for the pattern [<PERSON> . .. <JOB-SITUATION-WORD> ...<POST>+] . When matched against the sentence &quot;Mr . James, 57 years old, is stepping down as chief executive officer on July 1 and will retire as chairman at the end of the year .&quot;, &quot;James&quot; would be <PERSON>, &quot;stepping down&quot; would be <JOB-SITUATION-WORD> and &quot;chief executive officer&quot; would be <POST>.</Paragraph> <Paragraph position="6"> The semantic lexicon is separate from the parser's lexicon and has much less coverage . We use multiple levels of semantic lexicons: first, a generic application-independent lexicon with very shallow semantic information, then th e TE lexicon which provides more detailed entity-related semantic information, and finally the ST-level lexicon whic h provides detailed succession-related entries . Lexical semantic entries indicate the word's semantic type (a domain model concept), as well as predicates pertaining to it . For example, here is the lexical semantic entry for &quot;step In Figure 4, we show the semantic representation that is built for the sentence &quot;He will be succeeded by Mr . Dooner, 45 .&quot; in the walkthrough article. The whole sentence was parsed as a single fragment by FPP. The YOB-SITUATION event is a flat object combining information from both SUCCESSION and IN-AND-OUT objects for ease of processing. The POSmON-STATUS-GEN-IN state of affairs indicates that the system is unsure of whethe r the status should be &quot;IN&quot; or &quot;IN_ACTING&quot; . Note the link between the &quot;Unknown role&quot; in the JOB-SITUATION and the PERSON &quot;He&quot;. The discourse component will resolve the reference for the pronoun and will further refin e the relationship between the PERSON and the JOB-SITUATION .</Paragraph> <Paragraph position="7"> Figure 4 . Semantic Structure : The semantic representation for &quot;He will be succeeded by Mr . Dooner, 45. &quot; PLUM's discourse component creates a meaning for the whole message from the meaning of each sentence. The message level representation is a list of discourse domain objects (DDOs) for the top-level events of interest in the message (e.g., SUCCESSION events in the ST domain) . The semantic representation of a phrase in the text only includes information contained nearby ; the discourse module must infer other long-distance or indirect relations no t explicitly found by the semantic interpreter and resolve any references in the text .</Paragraph> <Paragraph position="8"> The discourse component creates two primary structures : a discourse predicate database and the DDOs . The database contains all the predicates mentioned in the semantic representation of the message . When references are resolved, corresponding semantic variables are unified . Any other inferences are also added to the database.</Paragraph> <Paragraph position="9"> To create the DDOs, the discourse component processes each semantic form produced by the interpreter, adding it s information to the database . It performs reference resolution for pronouns and anaphoric definite NPs ; set- and member-type references may be treated. The discourse component then applies inference rules that may add more semantic information to the discourse predicate database . When a semantic form for an event of interest i s encountered, a DDO is generated and any slots already found by the interpreter are filled in. The discourse processo r then tries to merge the new DDO with a previous DDO, in order to account for the possibility that the new DD O might be a repeated reference to an earlier one .</Paragraph> <Paragraph position="10"> Once all the semantic forms have been processed, heuristic rules are applied to fill any empty slots from the text surrounding the forms that triggered a given DDO . Each filler found in the text is assigned a confidence score base d on distance from trigger. Fillers found nearby are of high confidence, while those farther away receive worse score s (low numbers represent high confidence ; high numbers low confidence ; thus 0 is the &quot;highest&quot; confidence score) . In the ST system, the discourse processor performs an additional task. In order to simplify intermediate processing (semantic interpretation, DDO merging, pattern matching), a flat DDO, JOB-SITUATION, was defined whic h contains the equivalent of the information in a particular SUCCESSION/IN-AND-OUT pair . The flat structure of the JOB-SITUATION object makes merging much simpler as well as making the case frame and semantic pattern s easier to define. These must be converted to SUCCESSION and IN-AND-OUT objects before the template generation step, since this is what the template generator expects . The discourse processor performs this conversion . Each trigger fragment contains one or more words whose semantics triggered the DDO. A DDO can have multiple trigger fragments if the discourse component determines that the triggers co-refer . In this example, &quot;stepping down&quot; in the first fragment and &quot;succeeded&quot; in the second fragment are judged by the discourse processor to be referring t o the same succession . A score of 0 indicates that the filler was found either directly by the semantics or by a sentence-level pattern ; 1 that it was found in the same fragment as a trigger form ; 2 in the same sentence ; 4 in the The template generator takes the DDOs produced by discourse processing and fills out the application-specifi c templates. Clearly, much of this process is governed by the specific requirements of the application, considerations which have little to do with linguistic processing . The template generator must address any arbitrary constraints, a s well as deal with the basic details of formatting .</Paragraph> <Paragraph position="11"> The template generator uses a combination of data-driven and expectation-driven strategies . First the DDOs found by the discourse module are used to produce template objects . Next, the slots in those objects are filled usin g information in the DDO, the discourse predicate database, other sources of information such as the message heade r (e.g ., document number), or from heuristics (e .g., the type of an organization object is most likely to be COMPANY). The template definitions for the objects which are common to both TE and ST are almost identical .</Paragraph> <Paragraph position="12"> However, they sometimes differ due to the fact that different heuristics or template filling strategies may result i n better performance in each of the domains .</Paragraph> <Paragraph position="13"> Parameters in PLUM Many aspects of PLUM's behavior can be controlled by simply varying the values of system parameters . For example, PLUM has parameters to control aspects of tagging, parsing, pattern matching, event merging and slo t filling by discourse, and template filling . An important goal has been to make our system as &quot;parameterizable&quot; a s possible, so that the same software can meet different demands for recall, precision, and overgeneration . The final parameter settings for the test were generated by running the systems over all of the data we had and choosing the setting which seemed to maximize the value of the F-measure .</Paragraph> </Section> </Section> <Section position="6" start_page="61" end_page="62" type="metho"> <SectionTitle> TRAINING TECHNIQUE S </SectionTitle> <Paragraph position="0"> Half of the training data was set aside for blind test until the last week of the evaluation ; the remaining half was for development. This avoids overfitting to the development data. Since so little data was available, we also created our own training data from Wall Street Journal articles from 1987-1992 . This was created by retrieving article s using the University of Massachusetts document retrieval engine INQUERY . This gave us more training data (though presumably of a lesser quality) without violating the integrity of our high-quality blind test set . The blind test set was used to measure our progress at least once a week, with the frequency increasing as the end of the evaluation approached . Figure 5 shows how performance on the ST task improved over time on our blind test set .</Paragraph> <Paragraph position="1"> Early on, we identified key sentences (those sentences directly responsible for the generation of some importan t entity such as a SUCCESSION) . This allowed us to focus development on those portions of the data which wer e directly relevant to the task without having to always read through the irrelevant portions .</Paragraph> <Paragraph position="2"> We also evaluated system changes on a daily basis using the scores from the training/development set . The scoring program served as a guide for our development . Also, the message-by-message output allowed us to zero i n on messages where our performance was particularly bad and allowed us to add lexical items or semanti c interpretation rules based on the key sentences in the message . Since the new scoring software did not suppor t visualization of the differences between system output and the answer key, we wrote a visualization tool to do so . In all evaluation tasks (NE, TE, and ST), PLUM was run over all messages to detect and correct any causes o f system breaks. During the last week, we &quot;opened&quot; the blind material which had been released in early September . The entity name slot for all messages was used to quickly add names to the domain-dependent lexicon for TE and ST .</Paragraph> <Paragraph position="3"> The texts were examined for words or collocations which needed to be added to the domain-dependent lexicon . An automated tool generated n-tuples based on part-of-speech tag to aid this process.</Paragraph> <Paragraph position="4"> TE was slightly different as the training data from the dry run was still valid. Despite this, once the new training data arrived we concentrated almost exclusively on it, mainly using the older data as a &quot;sanity check&quot; before making system changes. We believed that this was important because the nature of the messages from the dry run was quit e different than that of the test messages because they had been chosen based on their relevance to the ST domain .</Paragraph> <Paragraph position="5"> A FUNNY THING HAPPENED ON THE WAY TO MUC- 6 Early in the planning of MUC-6, an additional dimension for evaluating parsers was planned . We prepared SPATTER for such an evaluation, and had achieved quite high scores on blind test material . However, the parsing evaluation was eventually canceled after the dry run for MUC-6, held in April-May .1 At that point, we decided that the limited resources we had for MUC should be devoted exclusively to improve scores on the application tasks (NE , TE, and ST), rather than trying to integrate the SPATTER parser for the application evaluations .</Paragraph> <Paragraph position="6"> We look forward to integrating SPATTER in future information extraction tasks to test the hypothesis that a far more accurate parser could lead to more accurate understanding and to notably higher scores .</Paragraph> </Section> <Section position="7" start_page="62" end_page="62" type="metho"> <SectionTitle> LIMITING FACTORS </SectionTitle> <Paragraph position="0"> Three factors significantly limited us. The first was the relative lack of data provided. Providing twice the messages marked for NE, TE, and ST would have made a big difference . Consequently, we created our ow n additional data and answer keys for NE and ST . Given so few messages, we felt that there were conventions in filling out the keys in each task that were still not fully clear . In fact, with students experienced in marking NE , consistency across annotators was only 94%, suggesting that the annotation rules can use further elaboration .</Paragraph> <Paragraph position="1"> Due to ongoing application efforts with tight deadlines, the limited availability of experienced MUCcaneers, an d prior investment in software to find names, we put in less effort than on any of the MUCs (-3, -4, and -5) which we had previously participated in .</Paragraph> <Paragraph position="2"> As a consequence of needing to limit the effort that we could give, we decided to focus on ST more than the other two tasks. TE received the least effort.</Paragraph> <Paragraph position="3"> 1. The most exciting is that near human performance is within the state of the art for mixed cas e English. Several systems performed at 90% or above .</Paragraph> <Paragraph position="4"> 2. Our next steps are to improve IdentiFinder's prediction of aliases once a name has been seen and to add rules fo r low frequency cases, e .g., improving performance on names that are quite unlike Western European names . 3. We would encourage looking at harder cases for NE evaluation . In Wall Street Journal, NE i s substantially simplified by accurate usage of mixed case . How would these systems perform in upper case only or in languages where initial capitalization does not signal a name? Languages such as Japanese and Chines e have no capital letters ; languages such as German use capitalization for all nouns, not just nouns in names .</Paragraph> <Section position="1" start_page="62" end_page="62" type="sub_section"> <SectionTitle> Regarding TE </SectionTitle> <Paragraph position="0"> We conclude the following : 1. As with NE, many groups performed at a level higher than any previous template fill task i n MUC-3, -4, or -5 . It will be interesting to see if this general template task is broadly useful, and whethe r performance is at a level high enough to warrant deployment in some real task(s). 1 Given the already ambitious nature of MUC-6, we do not disagree with the decision to consolidate on the fou r evaluation tasks, nor are we arguing to make parsing an evalution task in MUC-7 . 6 3 2. Given the little effort we invested in TE, we believe that another two person weeks could get PLUM's scores on blind test material over 80. To do this, we would improve NE performance (see discussion of NE above) an d would work further on the locale, country, and descriptor slots. An example of simple alias improvements is to recognize &quot;Mr . Smith&quot; as an alias after seeing &quot;John Smith,&quot; instead of merely predicting &quot;Smith&quot; and &quot;John&quot; a s possible aliases . We also believe that it is possible to achieve an F of 80 or better in TE using only lightweigh t processing.</Paragraph> <Paragraph position="1"> 3. To achieve an F above 90, on the other hand, is likely to require significant overall improvement and heavyweight processing in particular . We would incorporate the SPATTER parser, which parses far mor e accurately than FPP does, and would look to aid to our domain-independent semantic lexicon, so that there i s more semantic information to support merging of entity descriptions . Together, we believe these offer the best prospect for radically improved performance in the descriptor, locale and country slots .</Paragraph> <Paragraph position="2"> 4. Our TE system by design, employs no domain-specific knowledge . It therefore should work equally well on other text, not specific to change in corporate officers .</Paragraph> <Paragraph position="3"> 5. PLUM's Performance had not peaked on TE, since we put the least effort on it of any of the evaluatio n dimension . Furthermore, the official test was not even reflective of PLUM's performance, since a set of rule variations that was known to improve performance was saved at the ST level, rather than at the TE level (whic h 1. Compared to PLUM's previous performance in MUC-3, -4, and -5, our progress was muc h more rapid and our official score was higher than in any previous template fill task.</Paragraph> <Paragraph position="4"> 2. The division of objects into broadly applicable ones (TE) and domain-specific ones (ST) was a plus in ou r opinion. It felt as though the amount of domain-specific overhead was lowered, compared to previous MUCs . 3. Given that, we urge an even simpler template structure for future MUCs, one where only two levels are present : entities and relationships. This would more closely match what is stated in the text and would factor out th e .issues of data base structure . Together that should reduce overhead for participants further .</Paragraph> <Paragraph position="5"> 4. Though we believe that an additional 5-10 point improvement in F would have been achievable with mor e calendar time than 30 days for the ST task, to achieve an F above 65 is likely to require significant overall improvement . We would like to incorporate the SPATTER parser, which parses far more accurately than FP P does, and would look to add to our domain-independent semantic lexicon so as to improve merging of entit y descriptions. Together, this should improve overall recognition, merging and discrimination of all objects, an d may be a key to accurately recognizing events/relationships not detectable in a single sentence .</Paragraph> </Section> </Section> <Section position="8" start_page="62" end_page="64" type="metho"> <SectionTitle> OVERALL CONCLUSION S </SectionTitle> <Paragraph position="0"> We make two general observations . First, the state of the art has progressed greatly in portability in the last fou r years. For MUC-3, some high performing groups invested a small number of person years . By contrast, several groups this year achieved an F in the 50s in 30 calendar days .</Paragraph> <Paragraph position="1"> Yet, we believe that we are only beginning to understand techniques for learning domain-independent knowledg e and domain-dependent knowledge . Far more can be achieved . BBN particularly would like to investigate ho w statistical algorithms over large unmarked corpora can effectively extrapolate from a few training examples, such a s in ST in MUC-6, to provide greater coverage. For example, statistical techniques may have suggested th e importance of &quot;hire,&quot; a verb which many groups did not define .</Paragraph> <Paragraph position="2"> Second, since there has been a marked improvement in the quality of full parsers, now achieving an F in the hig h 80s (Magerman, 1995), we believe it is now feasible to consider using full parsers again . The rationale is straightforward : for full templates (e .g., ST) scores have been mired with an F in the 50s ever since MUC-3 in 1991 . Pattern matching has given us very robust, very portable technology, but has not broken the performance barrier al l systems have run up against. Recent (statistical) full parsers (e.g., BBN's, IBM's, and UPenn's) have such quantitatively better performance that they are qualitatively better . We believe this offers the opportunity to agai n try heavyweight techniques to attempt deeper understanding . Pattern matching techniques will still have a crucia l role for domain-specific details, but we believe they can be greatly improved by deeper understanding .</Paragraph> </Section> class="xml-element"></Paper>