<?xml version="1.0" standalone="yes"?>
<Paper uid="A00-2043">
  <Title>An Empirical Assessment of Semantic Interpretation</Title>
  <Section position="3" start_page="0" end_page="329" type="metho">
    <SectionTitle>
2 The Basic Model for Semantic
</SectionTitle>
    <Paragraph position="0"> Interpretation The problem of semantic interpretation can be described as the mapping from syntactic to semantic (or conceptual) representation structures. In our approach, the syntactic representation structures are given as dependency graphs (Hahn et al., 1994). Unlike constituency-based syntactic descriptions, dependency graphs consist of lexical nodes only, and these nodes are connected by vertices, each one of which is labeled by a particular dependency relation (cf. Figure 1).</Paragraph>
    <Paragraph position="1"> For the purpose of semantic interpretation, dependency graphs can be decomposed into semantically interpretable subgraphs3 Basically, two types of semantically interpretable subgraphs can be distinguished. The first one consists of lexical nodes which are labeled by content words only (lexical instances of verbs, nouns, adjectives or adverbs) and which are directly linked by a single dependency relation of any type whatsoever. Such a subgraph is illustrated in Figure 1 by 8peicher- genatt - Computers. The second type of subgraph is also delimited by labels of content words but, in addition, a series of n -- 1... 4 intermediary lexical nodes may 1This notion and all subsequent criteria for interpretation are formally described in Romacker et al. (1999).</Paragraph>
    <Paragraph position="2">  appear between these content words, all of which are labeled by non-content words (such as auxiliary or modal verbs, prepositions). Hence, in contrast to direct linkage we speak here of indirect linkage between content words. Such a subgraph, with two intervening non-content words - the modal &amp;quot;kann&amp;quot; and the auxiliary &amp;quot;werden&amp;quot; -, is given in Figure 1 by Speieher- subject - kann- verbpart - werdenverbpart - erweitert. Another subgraph with just one intervening non-content word - the preposition &amp;quot;mit&amp;quot; - is illustrated by erweitert- ppadjunct - mit - pobject - SDRAM-Modulen. From these considerations follows that, e.g., the subgraph spanned by Speieher and SDRAM-Modulen does not form a semantically interpretable subgraph, since the content word erweitert intervenes on the linking path.</Paragraph>
    <Paragraph position="3"> Our approach to semantic interpretation subscribes to the principles of locality and compositionality. It operates on discrete and well-defined units (subgraphs) of the parse tree, and the results of semantic interpretation are incrementally combined by fusing semantically interpretable subgraphs.</Paragraph>
    <Paragraph position="4"> As semantic target language we have chosen the framework of KL-ONE-type description logics (DL) (Woods and Schmohe, 1992). Since these logics are characterized by a settheoretical semantics we stay on solid formal ground. Fhrthermore, we take advantage of the powerful inference engine of DL systems, the description classifier, which turns out to be essential for embedded reasoning during the semantic interpretation process. By equating the semantic representation language with the conceptual one, we follow arguments discussed by Allen (1993).</Paragraph>
    <Paragraph position="5"> The basic idea for semantic interpretation is as follows: Each lexical surface form of a content word is associated with a set of concept identifiers representing its (different) lexical meanings. This way, lexical ambiguity is accounted for. These conceptual correlates are internal to the domain knowledge base, where they are described by a list of attributes or conceptual roles, and corresponding restrictions on permitted attribute values or role fillers are associated with them.</Paragraph>
    <Paragraph position="7"> As an example, consider the description for the concept COMPUTER-SYSTEM. It may be characterized by a set of roles, such as HAS-HARD-DISK Or HAS-WORKING-MEMORY, with corresponding restrictions on the concept types of potential role fillers. HAS-WORKING-MEMORY, e.g., sanctions only fillers of the concept type MEMORY. These conceptual constraints are used for semantic filtering, i.e., for the elimination of syntactically admissible dependency graphs which, nevertheless, do not have a valid semantic interpretation.</Paragraph>
    <Paragraph position="8"> Semantic interpretation, in effect, boils down to finding appropriate conceptual relations in the domain knowledge that link the conceptual correlates of the two content words spanning the semantically interpretable subgraph, irrespective of whether a direct or an indirect linkage holds at the syntactic level. Accordingly, Figure 2 depicts the semantic/conceptual interpretation of the dependency structure given in Figure 1. Instances representing the concrete discourse entities and events in the sample sentence are visualized as solid rectangles containing a unique identifier (e.g., COMPUTER-SYSTEM.02). Labeled and directed edges indicate instance roles. Dashed rectangles characterize symbols used as makers for tense and modality. 2 Note that in Figure 2 each tuple of content words which configures a minimal subgraph in Figure 1 has already received an interpretation in terms of a relation linking the conceptual correlates. For example, Speicher- genatt - Computers (cf. Figure 1, box 1) is mapped to COMPUTER-SYSTEM.02 HAS-WORKING-MEMORY MEMORY.01 (cf. Figure 2, box 1). However, the search for a valid conceptual relation is not only limited to a simple one-link slot-filler structure. We rather may determine conceptual relation paths between conceptual correlates of lexical items, the length of which may be greater than 1.</Paragraph>
    <Paragraph position="9">  (Thus, the need for role composition in the DL language becomes evident.) The directed search in the concept graph of the domain knowledge requires sophisticated structural and topological constraints to be manageable at all. These constraints are encapsulated in a special path finding and path evaluation algorithm specified in Markert and Hahn (1997).</Paragraph>
    <Paragraph position="10"> Besides these conceptual constraints holding in the domain knowledge, we further attempt to reduce the search space for finding relation paths by two kinds of syntactic criteria. First, the search may be constrained by the type of dependency relation holding between the content words of the currently considered semantically interpretable subgraph (direct linkage), or it may be constrained by the intervening lexical material, viz. the non-content words (indirect linkage). Each of these syntactic constraints has an immediate mapping to conceptual ones.</Paragraph>
    <Paragraph position="11"> For some dependency configurations, however, no syntactic constraints may apply. Such a case of unconstrained semantic interpretation (e.g., for genitive attributes directly linked by the genatt relation) leads to an exhaustive directed search in the knowledge base in order to find all conceptually compatible role fillings among the two concepts involved. Syntactic restrictions on semantic interpretation either come from lexeme classes or concrete lexemes.</Paragraph>
    <Paragraph position="12"> They are organized in terms of the lexeme class hierarchy superimposed on the fully lexicalized dependency grammar we use (Hahn et al., 1994). In the fragment depicted in Figure 3, the lexeme class of transitive verbs, VERBTRANS, requires that whenever a subject dependency relation is encountered, semantic interpretation is constrained to the conceptual roles AGENT or PATIENT and all their subrelations (such as EXTENSION-PATIENT). All other conceptual roles are excluded from the subsequent semantic interpretation. Exploiting the property inheritance mechanisms provided by the hierarchic organization of the lexicalized dependency grammar, all concrete lexemes subsumed by the lexeme class VERBTRANS, like &amp;quot;erweitern&amp;quot; (extend), inherit the corresponding constraint. However, there are lexeme classes such as NOUN which do not render any constraints for dependency relations such as evidenced by gen\[itive\] att\[ribute\] (cf. Fig. 3).</Paragraph>
    <Paragraph position="13"> It may even happen that such restrictions can only be attached to concrete lexemes in order to avoid overgeneralization. Fortunately, we observed that this only happened to be the case for closed-class, i.e., non-content words. Accordingly, in Figure 3 the preposition &amp;quot;with&amp;quot; is characterized by the constraint that only the conceptual roles HAS-PART, IN-STRUMENT, etc. must be taken into consideration for semantic interpretation.</Paragraph>
    <Paragraph position="14"> Since the constraints at the lexeme class or the lexeme level are hard-wired in the class hierarchy, we refer to the mapping of dependency relations (or idiosyncratic lexemes) to a set of conceptual relations (expanded to their transitive closure) as static interpretation. In contradistinction, the computation of relation paths for tuples of concepts during the sentence analysis process is called dynamic interpretation, since the latter process incorporates additional conceptual constraints on the fly.</Paragraph>
    <Paragraph position="15"> The above-mentioned conventions allow the specification of high-level semantic interpretation schemata covering a large variety of different syntactic constructions by a single schema. For instance, each syntactic construction for which no conceptual constraints apply (e.g., the interpretation of genitives, most adjectives, etc.) receives its semantic interpretation by instantiating the same interpretation schema (Romacker et al., 1999). The power of this approach comes from the fact that these high-level schemata are instantiated in the course of the parsing process by exploiting the dense specifications of the inheritance hierarchies both at the grammar level (the lexeme class hierarchy), as well as the conceptual level (the concept and role hierarchies).</Paragraph>
    <Paragraph position="16"> We currently supply up to ten semantic interpretation schemata for declaratives, relatives, and pas- null sives at the clause level, complement subcategorization via PPs, auxiliaries, all tenses at the VP level, pre- and and postnominal modifiers at the NP level, and anaphoric expressions. We currently do not account for control verbs (work in progress), coordination and quantification.</Paragraph>
  </Section>
  <Section position="4" start_page="329" end_page="330" type="metho">
    <SectionTitle>
3 The Evaluation of Semantic
</SectionTitle>
    <Paragraph position="0"> Interpretation In this section, we want to discuss, for two particular types of German language phenomena, the adequacy of our approach in the light of concrete language data taken from the two corpora we work with. This part of the enterprise, the empirical assessment of semantic interpretation, is almost entirely neglected in the literature (for two notable exceptions, cf. Bonnema et al. (1997) and Bean et al. (1998)).</Paragraph>
    <Paragraph position="1"> Though similarities exist (viz. dealing with the performance of NLP systems in terms of their ability to generate semantic/conceptual structures), the semantic interpretation (SI) task has to be clearly distinguished from the information extraction (IE) task and its standard evaluation settings (Chinchor et al., 1993). In the IE task, a small subset of the templates from the entire domain is selected into which information from the texts are mapped. Also, the design of these templates focus on particularly interesting facets (roles, in our terminology), so that an IE system does not have to deal with the full range of qualifications that might occur -- even relating to relevant, selected concepts. Note that in any case, a priori relevance decisions limit the range of a posteriori fact retrieval.</Paragraph>
    <Paragraph position="2"> The SI task, however, is far less restricted. We here evaluate the adequacy of the conceptual representation structures relating, in principle (only restricted, of course, by the limits of the knowledge acquisition devices), to the entire domain of discourse, with all qualifications mentioned in a text. Whether these are relevant or not for a particular application has to be determined by subsequent data/knowledge cleansing. In this sense, semantic interpretation might deliver the raw data for transformation into appropriate IE target structures. Only because of feasibility reasons, the designers of IE systems equate IE with SI. The cross-linking of IE and SI tasks, however, bears the risk of having to determine, in advance, what will be relevant or not for later retrieval processes, assumptions which are likely to be flawed by the dynamics of domains and the unpredictability of the full range of interests of prospective users.</Paragraph>
    <Section position="1" start_page="329" end_page="330" type="sub_section">
      <SectionTitle>
3.1 Methodological Issues
</SectionTitle>
      <Paragraph position="0"> Our methodology to deal with the evaluation of semantic interpretation is based on a triple division of test conditions. The first category relates to checks whether so-called static constraints, effected by the mapping from a single dependency relation to one or more conceptual relations, are valid (cf. Figure 3 for restrictions of this type). Second, one may investigate the appropriateness of the results from the search of the domain knowledge base, i.e., whether a relation between two concepts can be determined at all, and, if so, whether that relation (or role chain) is adequate. The conceptual constraints which come into play at this stage of processing are here referred to as dynamic constraint propagation, since they are to be computed on the fly, while judging the validity of the role chain in question. 3 Third, interactions between the above-mentioned static constraints and dynamic constraint propagation may occur. This is the case for the interpretation of auxiliaries or prepositions, where intervening lexical material and associated constraints have to be accounted for simultaneously. null In our evaluation study, we investigated the effects of category II and category III phenomena by considering genitives and modal as well as auxiliary verbs, respectively. The knowledge background is constituted by a domain ontology that is divided into an upper generic part (containing about 1,500 concepts and relations) and domain-specific extensions. We here report on the two specialized domains we deal with -- a hardware-biased information technology (IT) domain model and an ontology covering parts of anatomical medicine (MED). Each of these two domain models adds roughly about 1,400 concepts and relations to the upper model. Corresponding lexeme entries in the lexicon provide linkages to the entire ontology. In order to avoid error chaining, we always assume a correct parse to be delivered for the semantic interpretation process.</Paragraph>
      <Paragraph position="1"> We took a random selection of 54 texts (comprising 18,500 words) from the two text corpora, viz. IT test reports and MEDical finding reports. For evaluation purposes (cf. Table 1), we concentrated on the interpretation of genitives (as an instance of direct linkage; GEN) and on the interpretation of periphrastic verbal complexes, i.e., passive, temporal and modal constructions (as instances of indirect linkage; MODAUX).</Paragraph>
      <Paragraph position="2"> The choice of these two grammatical patterns allows us to ignore the problems caused by syntactic ambiguity, since in our data no structural am3Note that computations at the domain knowledge level which go beyond mere type checking are usually located outside the scope the semantic considerations. This is due to the fact that encyclopedic knowledge and its repercussions on the understanding process are typically not considered part of the semantic interpretation task proper. While this may be true from a strict linguistic point of view, from the computational perspective of NLP this position cannot seriously be maintained. Even more so, when semantic and conceptual representations are collapsed.</Paragraph>
      <Paragraph position="3">  biguities occurred. If one were to investigate the combined effects of syntactic ambiguity and semantic interpretation the evaluation scenario had to be changed. Methodologically, the first step were to explore the precision of a semantic interpretation task without structural ambiguities (as we do) and then, in the next step, incorporate the treatment of syntactic ambiguities (e.g., by semantic filtering devices, cf. Bonnema et al. (1997)).</Paragraph>
      <Paragraph position="4"> Several guidelines were defined for the evaluation procedure. A major issue dealt with the correctness of a semantic interpretation. In cases with interpretation, we considered a semantic interpretation to be a correct one, if the conceptual relation between the two concepts involved was considered adequate by introspection (otherwise, incorrect). This qualification is not as subjective as it may sound, since we applied really strict conditions adjusted to the fine-grained domain knowledge. 4 Interpretations were considered to be correct in those cases which contained exactly one relation, as well as cases of semantical/conceptual ambiguities (up to three readings, the most), presumed the relation set contained the correct one. 5 A special case of incorrectness, called nil, occurred when no relation path could be determined though the two concepts under scrutiny were contained in the domain knowledge base and an interpretation should have been computed.</Paragraph>
      <Paragraph position="5"> We further categorized the cases where the system failed to produce an interpretation due to at least one concept specification missing (with respect to the two linked content words in a semantically interpretable subgraph). In all those cases without interpretation, insufficient coverage of the upper model was contrasted with that of the two domain models in focus, MED and IT, and with cases in which concepts referred to other domains, e.g., fashion or food. Ontological subareas that could neither be assigned to the upper model nor to particular domains were denoted by phrases referring to time (e.g., &amp;quot;the beginning of the year&amp;quot;), space (e.g.,  &amp;quot;the infiltration of the stroma&amp;quot; resulted in a correct reading - STROMA being the PATIENT of the INFILTRATION event -, as well as in an incorrect one - being the AGENT of the IN-FILTRATION. Among the incorrect semantic interpretations we also categorized, e.g., the interpretation of the expression &amp;quot;the prices of the manufacturers&amp;quot; as a conceptual linkage from PRICE via PRICE-OF to PRODUCT via HAS-MANUFACTURER to MANUFACTURER (this type of role chaining can be considered an intriguing example of the embedded reasoning performed by the description logic inference engine), since it did not account for the interpretation that MANUFACTURERS fix PRICES as part of their marketing strategies. After all, correct interpretations always boiled down to entirely evident cases, e.g.,</Paragraph>
    </Section>
  </Section>
  <Section position="5" start_page="330" end_page="332" type="metho">
    <SectionTitle>
HARD-DISK PART-OF COMPUTER.
</SectionTitle>
    <Paragraph position="0"> 5At the level of semantic interpretation, the notion of semantic ambiguity relates to the fact that the search algorithm for valid conceptual relation paths retrieves more than a single relation (chain).</Paragraph>
    <Paragraph position="1"> &amp;quot;the surface of the storage medium&amp;quot;), and abstract notions (e.g., &amp;quot;the acceptance of IT technology&amp;quot;). Finally, we further distinguished evaluative expressions (e.g., &amp;quot;the advantages of plasma display&amp;quot;) from figurative language, including idiomatic expressions (e.g., &amp;quot;the heart of the notebook&amp;quot;).</Paragraph>
    <Paragraph position="2"> At first glance, the choice of genitives may appear somewhat trivial. From a syntactic point of view, genitives are directly linked and, indeed, constitute an easy case to deal with at the dependency level.</Paragraph>
    <Paragraph position="3"> From a conceptual perspective, however, they provide a real challenge. Since no static constraints are involved in the interpretation of genitives (cf. Figure 3, lexeme class NOUN) and, hence, no prescriptions of (dis)allowed conceptual relations are made, an unconstrained search (apart from connectivity conditions imposed on the emerging role chains) of the domain knowledge base is started. Hence, the main burden rests on the dynamic constraint processing part of semantic interpretation, i.e., the path finding procedure muddling through the complete domain knowledge base in order to select the adequate conceptual reading(s). Therefore, genitives make a strong case for test category II mentioned above.</Paragraph>
    <Paragraph position="4"> Dependency graphs involving modal verbs or auxiliaries are certainly more complex at the syntactic level, since the corresponding semantically interpretable subgraphs may be composed of up to six lexical nodes. However, all intervening noncontent-word nodes accumulate constraints for the search of a valid relation for semantic interpretations and, hence, allows us to test category III phenomena. The search space is usually pruned, since only those relations that are sanctioned by the intervening nodes have to be taken into consideration.</Paragraph>
    <Section position="1" start_page="330" end_page="332" type="sub_section">
      <SectionTitle>
3.2 Evaluation Data
</SectionTitle>
      <Paragraph position="0"> We considered a total of almost 250 genitives in all these texts, from which about 59%/33% (MED/IT) received an interpretation. 6 Out of the total loss due to incomplete conceptual coverage, 56%/58% (23 of 41 genitives/57 of 98 genitives) can be attributed to insufficient coverage of the domain models. Only the remaining 44%/42% are due to the residual factors listed in Table 1.</Paragraph>
      <Paragraph position="1"> In our sample, the number of syntactic constructions containing modal verbs or auxiliaries amout to 292 examples. Compared to genitives, we obtained a more favorable recall for both domains: 66% for MED and 40% for IT. As for genitives, lacking interpretations, in the majority of cases, can be attributed to insufficient conceptual coverage. For the IT domain, however, a dramatic increase in the number of missing concepts is due to gaps in the upper model (78 or 63%) indicating that a large number of 6Confidence intervals at a 95% reliability level are given in brackets in Table 1.</Paragraph>
      <Paragraph position="2">  iliaries (MODAUX) in the IT and MED domains essential concepts for verbs were not modeled. Also, figurative speech plays a more important role in IT with 24 occurrences. Both observations mirror the fact that IT reports are linguistically far less constrained and are rhetorically more advanced than their MED counterparts.</Paragraph>
      <Paragraph position="3"> Another interesting observation which is not made explicit in Table 1 concerns the distribution of modal verbs and auxiliaries. In MED, we encountered 57 passives and just one modal verb and no temporal auxiliaries, i.e., our data are in line with prevailing findings about the basic patterns of medical sublanguage (Dunham, 1986). For the IT domain, corresponding occurrences were far less biased, viz. 80 passives, 131 modal verbs, and 23 temporal auxiliaries. Finally, for the two domains 25 samples contained both modal verbs and auxiliaries, thus forming semantically interpretable subgraphs with four word nodes.</Paragraph>
      <Paragraph position="4"> One might be tempted to formulate a null hypothesis concerning the detrimental impact of the length of semantically interpretable subgraphs (i.e., the number of intervening lexical nodes carrying non-content words) on the quality of semantic interpretation. In order to assess the role of the length of the path in a dependency graph, we separately investigated the results for these subclasses of combined verbal complexes* From the entire four-node set (cf. Table 2) with 25 occurrences (3 for MED and 22 for IT), 16 received an interpretation (3 for MED, 13 for IT). While we neglect the MED data due to the small absolute numbers, the IT domain revealed  59% recall and 85% precision. If we compare this to the overall figures for recall (40%) and precision (85%), the data might indicate a gain in recall for longer subgraphs, while precision keeps stable.</Paragraph>
      <Paragraph position="5"> The results we have worked out are just a first step into a larger series of broader and deeper evaluation efforts. The concrete values we present, sobering as they may be for recall (57%/31% for genitives and 66%/40% for modal verbs and auxiliaries), encouraging, however, for precision (97%/94% for genitives and 95%/85% for modal verbs and auxiliaries), can only be interpreted relative to other data still lacking on a broader scale.</Paragraph>
      <Paragraph position="6"> As with any such evaluation, idiosyncrasies of the coverage of the knowledge bases are inevitably tied with the results and, thus, put limits on too far-reaching generalizations. However, our data reflect the intention to submit a knowledge-intensive text understander to a realistic, i.e., conceptually unconstrained and therefore &amp;quot;unfriendly&amp;quot; test environment. null  Judged from the figures of our recall data, there is no doubt, whatsoever, that conceptual coverage of the domain constitutes the bottleneck for any knowledge-based approach to NLP. ~ Sublanguage differences are also mirrored systematically in these data, since medical texts adhere more closely to well-established concept taxonomies and writing standards than magazine articles in the IT domain.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>