<?xml version="1.0" standalone="yes"?>
<Paper uid="W06-2610">
  <Title>An Ontology-Based Approach to Disambiguation of Semantic Relations</Title>
  <Section position="3" start_page="0" end_page="72" type="metho">
    <SectionTitle>
2 Semantic relations
</SectionTitle>
    <Paragraph position="0"> The following account is based on the work of (Jensen and Nilsson, 2006): Relations exist between entities referred to in discourse. They can exist at different syntactic levels; across sentence boundaries as in example 1, or within a sentence, a phrase or a word. The relations  can be denoted by different parts of speech, such as a verb, aprepositionoranadjective, ortheycanbeimplicitly present in compounds and genitive constructions as in example 2.</Paragraph>
    <Paragraph position="1"> Semantic relations are n-ary: In example 1 below the verb form 'owns' denotes a binary relation between Peter and a dog, and in example 3, the verb form 'gave' denotes a ternary relation between Peter, the dog and a bone. In example 4 the preposition 'in' denotes a binary relation between the dog and the yard.</Paragraph>
    <Paragraph position="2">  (1) Peter owns a dog. It is a German shepherd.</Paragraph>
    <Paragraph position="3"> (2) Peter's dog.</Paragraph>
    <Paragraph position="4"> (3) Peter gave the dog a bone.</Paragraph>
    <Paragraph position="5"> (4) The dog in the yard.</Paragraph>
    <Paragraph position="6">  In the framework of this machine learning project, we will only consider binary relations denoted by prepositions. A preposition, however, can be ambiguous in regard to which relation it denotes. As an example, let us consider the Danish preposition i (Eng: in): The surface form i in 'A i B' can denote at least five different  relations between A and B: 1. ApatientrelationPNT;arelationwhereoneofthe arguments' case role is patient, e.g. &amp;quot;aendringer i stofskiftet&amp;quot; (changes in the metabolism).</Paragraph>
    <Paragraph position="7"> 2. A locational relation LOC; a relation that denotes  thelocation/positionofoneoftheargumentscompared to the other argument, e.g. &amp;quot;skader i hjertemuskulaturen&amp;quot; (injuries in the heart muscle). 3. A temporal relation TMP; a relation that denotes the placement in time of one of the arguments compared to the other, e.g. &amp;quot;mikrobiologien i 1800-tallet&amp;quot; (microbiology in the 19th century). 4. A property ascription relation CHR; a relation that denotes a characterization relation between one of the arguments and a property, e.g. &amp;quot;antioxidanter irenfremstilletform&amp;quot;(antioxidantsinapureform) 5. A 'with respect to' relation WRT; an underspeci null fied relation that denotes an 'aboutness' relation between the arguments, e.g. &amp;quot;forskelle i saltindtagelsen&amp;quot; (differences in the salt intake) . As presented above, the idea is to perform supervised machine learning, that will take into account the surface form of the preposition and the ontological type of the heads of the surrounding noun phrases, and on this basis be able to determine the relation that holds between noun phrases surrounding a preposition in unseen text.</Paragraph>
  </Section>
  <Section position="4" start_page="72" end_page="74" type="metho">
    <SectionTitle>
3 The corpus
</SectionTitle>
    <Paragraph position="0"> In order to establish a training set, a small corpus of approximately 18,500 running words has been compiled from texts from the domain of nutrition and afterwards annotated with the ontological type of the head of the noun phrases, and the semantic relation denoted by the preposition 2.</Paragraph>
    <Paragraph position="1"> All the text samples in this corpus derive from &amp;quot;The DanishNationalEncyclopedia&amp;quot;(Gyldendal,2004),and are thus not only limited domain-wise, but also of a  veryspecifictexttypewhichcanbeclassifiedasexpertto-non-expert. Thus, we cannot be certain that our results can be directly transferred to a larger or more general domain, or to a different text type. This aspect would have to be empirically determined.</Paragraph>
    <Section position="1" start_page="72" end_page="74" type="sub_section">
      <SectionTitle>
3.1 Annotation
</SectionTitle>
      <Paragraph position="0"> For the purpose of learning relations, 952 excerpts of the form:</Paragraph>
      <Paragraph position="2"> information about part of speech, ontological type and relation type for NP heads and prepositions, respectively. An example of the analyzed text excerpts are given in table 1 on the following page, where each row indicates a level of the analysis.</Paragraph>
      <Paragraph position="3"> The POS-tagging and head extraction have been done automatically, the ontological type assignation partly automatically (ontology look-up) and partly manually (forwordsthatdonotexistasinstantiationsofconcepts in the ontology). The relation annotation has been done manually.</Paragraph>
      <Paragraph position="4"> The tags used in the annotation on the three levels are: POS-tags. Our tagger uses a subset of the PAROLE tag set, consisting of 43 tags, see (Hansen, 2000), which means that it is a low level POS tagging with little morphosyntactic information. We only use the tags in order to extract NPs and prepositions, and thus do not need a more fine-grained information level.</Paragraph>
      <Paragraph position="5"> SIMPLE-tags. The tags used for the ontological type annotation consist of abbreviations of the types in the SIMPLE top ontology. The tag set consists of 151 tags.</Paragraph>
      <Paragraph position="6"> Relation-tags. The tags used for the relation annotation derive from a minimal set of relations that have been used in earlier OntoQuery related work.</Paragraph>
      <Paragraph position="7"> The set can be seen in table 2 2Extraction, POS-tagging and initial ontological and relation type annotation was done by Dorte Haltrup Hansen,  surface form blodprop (thrombosis) i (in) hjertet (the heart) syntactic structure head of first NP preposition head of second NP relation and ontological type disease location body part  The manual relation annotation has been done by one annotator for this initial project. The ideal situation would be to have several annotators annotate the corpus. If two or more people annotate the same corpus, they are almost certain to disagree on some occasions. This disagreement can have two sources: first it can be due to cognitive differences. Two people subjected to the same utterance are not guaranteed to perceive the same content, or to perceive the content intended by the producer of the utterance. Many factors are at play here; cultural background, knowledge, memory, etc.</Paragraph>
      <Paragraph position="8"> Secondly, it can be due to conceptual, lexical or syntactic ambiguity in the utterance. We cannot remove these sources of disagreement, but we can introduce tools that make the annotation more consistent. By using a finite and minimal realtion tag set and, further, by introducing paraphrase tests, we hope to minimize the risk of inter-annotator disagreement in a future annotation on a larger scale.</Paragraph>
      <Paragraph position="9">  As noted above, the ontological types used in the experiments derive from the SIMPLE top ontology (Pedersen, 1999; Lenci et al., 2000). The heads of the phrases have been annotated with the lowest possible node, i.e. ontological type, of the top ontology. In the case of blodprop the annotation of ontological type is &amp;quot;disease&amp;quot;, since &amp;quot;disease&amp;quot; is the lowest node in the top ontology in the path from thrombosis to the top. This is illustrated in figure 1, which shows the path from blodprop (thrombosis) to the top level of SIMPLE.</Paragraph>
      <Paragraph position="10"> Thus, for the purpose of this project, we only consider one node for each concept: the lowest possible node in the top ontology. Another approach would be to consider the the full path to the top node, and also including the path from the leaf node to the lowest node in the top ontology. In the example depicted in figure 1, the full path from trombosis to the top node would be trombosis-cardiovascular diseasedisease-phenomenon-event-entity-top or trombosis-cardiovascular disease-disease-agentive-top.  For the purpose of the manual relation annotation, we neededtodecideonafinitesetofpossiblerelationsthat can be denoted by prepositions. This is a non-trivial task, as it is almost impossible to foresee which relations prepositions can denote generally, and in the text type at hand specifically, by introspection alone. The method that we decided to use was the following: An  (thrombosis) to the top level of the SIMPLE ontology.</Paragraph>
      <Paragraph position="11"> initial set of relations that have all been used in prior OntoQuery-related work (Nilsson, 2001; Madsen et al., 2001; Madsen et al., 2000), were chosen as a point of departure. The final set was found by annotating the text segments using this set as the possible relation types, and the relations that are actually manifested in the data then form the final subset that was used as input for a machine learning algorithm. The final subset  which is a subset of the set proposed in Nilsson, 2001.</Paragraph>
    </Section>
    <Section position="2" start_page="74" end_page="74" type="sub_section">
      <SectionTitle>
3.2 Paraphrase tests
</SectionTitle>
      <Paragraph position="0"> In order to ensure a consistent relation annotation, it is necessary to develop a set of paraphrase tests that can help the annotator determine which relation a given preposition denotes in a given context. Some relations are particularly difficult to intuitively keep apart from closely related relations. One of these problematic relation pairs is treated in some detail below.</Paragraph>
      <Paragraph position="1">  Forexamplelocativeandpartitiverelationscanbedifficulttokeepapart,probablybecausetheytosomeextent null are overlapping semantically. From a philosophical point of view, an important question is 'when does an entity become part of the entity it is located in?', but from a practical point of view, we are interested in answering the question 'how can we decide if a given relation a locative or partitive relation?'.</Paragraph>
      <Paragraph position="2"> In this paper we will only treat the latter question. A tool thatis useful for thispurpose is theparaphrase test: If we can paraphrase the text segment in question into the phrasing the test prescribes, while preserving the semantic content, we can conclude that the relation is a possible relation for the given phrase.</Paragraph>
      <Paragraph position="3">  The two relations LOC and POF can be difficult to differentiate, even when using paraphrase tests. Therefore, an additional test that could be considered, is Ruus' attribute transportation test (Ruus, 1995)3. In the example &amp;quot;The pages in the book&amp;quot;, the book gets e.g. the attribute 'binding: {hardback  |paperback}' from cover, and the attribute 'paper grade:{bond  |book  We cannot observe an attribute transport, neither from the bird to the roof, nor the other way. This suggests that it is possible to use the atrribute transportation test in order to determine whether a given relation is a POF or a LOC relation. Thus, we can now formulate the following paraphrase test for POF: POF: A consists e.g. of B and A has the attribute X, from B.</Paragraph>
    </Section>
  </Section>
  <Section position="5" start_page="74" end_page="75" type="metho">
    <SectionTitle>
4 Experiments
</SectionTitle>
    <Paragraph position="0"> The annotation process generates af a feature space of sixdimensions, namelythelemmatizedformofthetwo heads of the noun phrases, the ontological types of the heads, the preposition and the relation. In the corpus there is a total of only 952 text segments. In general the distribution of the data is highly skewed and sparseness is a serious problem. More than half of the instances are of the relation type WRT or PNT, and the rest of the instances are distributed among the remaining 10 relations with only 14 instances scattered over the tree smallest classes. This is illustrated in figure 3.</Paragraph>
    <Paragraph position="1"> There are 332 different combinations of ontological types where 197 are unique. There are 681 different heads and 403 of them are unique, with all of them being  possible relations.</Paragraph>
    <Paragraph position="2"> Our assumption is that there is consistency in which relations prepositions usually denote in particular contexts, and hence the learning algorithms should be able to generalize well. We also assume that the addition of the ontological types of the head of the NP, is the most vital information in classifying the relation type, at least in this case where data is sparse.</Paragraph>
    <Paragraph position="3"> We have run the experiments with a Support Vector Machine algorithm SMO (Keerthi et al., 2001) and the prepositional rule learning algorithm JRip (Cohen, 1995). The former in order to get high precision, the latter in order to get easily interpretable rules for later analysis (see section 4.1). The experiments were run using 10-fold-cross-validation, with a further partition of the training set at each fold into a tuning and a training set. The tuning set was used to optimize the parameter4 settings for each algorithm . The implementation of the algorithms that we used, was the WEKA software package (Frank et al., 2005).</Paragraph>
    <Paragraph position="4">  The experiments were run on seven different combinations of the feature space, ranging from using only the heads to using both heads, preposition and ontological types of the heads. This was done in order to get insight into the importance of using ontological types in the learning. The results of these experiments are shown in table 3. The last column shows the precision for a projected classifier (PC) in the cases where it out-performs the trivial rejector. The projected classifier, in this case, assigns the relation that is most common for the corresponding input pair; e.g if the ontological types are DIS/HUM, then the most common relation is PNT. The trivial rejector, which assigns the most common relation, in this case WRT, to all the instances, achieves a precision of 37.8%.</Paragraph>
    <Paragraph position="5">  classifier on the seven different combinations of input features. &amp;quot;Lemma&amp;quot; here is short for lemmatized NP head.</Paragraph>
    <Paragraph position="6"> The following conclusions can be drawn from table 3.</Paragraph>
    <Paragraph position="7"> The support vector machine algorithm produces a resultwhichinallcasesisbetterthanthebaseline, i.e. we are able to produce a model that generalizes well over the training instances compared to the projected classifier or the trivial rejector. This difference is not statistically significant at a confidence level of 0.95 when only training on the surface form of prepositions.</Paragraph>
    <Paragraph position="8"> A comparison of line 1-3 shows that training on ontological types seems to be superior to using lemmatized NP heads or prepositions, though the superiority is not statistically significant when comparing to the lemmatized NP heads. When comparing line 4-7 the difference between the results are not statistically significant. This fact may owe to the data sparseness. However, comparing line 1 to line 6 or 7, shows that the improvement of adding the preposition and the lemmatized NP heads to the ontological types is statistically significant.</Paragraph>
    <Paragraph position="9"> In general, the results reveal an unexplored opportunity to include ontological types and the relations that prepositions denote in information retrieval. In the next section, we will look more into the rules created by the JRip algorithm from a linguistic point of view.</Paragraph>
    <Section position="1" start_page="75" end_page="75" type="sub_section">
      <SectionTitle>
4.1 Analyzing the rules
</SectionTitle>
      <Paragraph position="0"> In this section we will take a deeper look into the rules produced by JRip on the data set with only ontological types,sincetheyarethemostinterestinginthiscontext.</Paragraph>
      <Paragraph position="1"> The JRip algorithm produced on average 21 rules. The most general rule covering almost half of the instances is the default rule, that assigns all instances to the WRT relation if no other rules apply. At the other end of the spectrum, there are ten rules covering no more than 34 instances, but with a precision of 100%. It is futile to analyse these rules, since they cover the most infrequent relations and hence may be overfitting the data set. However, this seems not be the case with a rule like &amp;quot;if the ontotype of the first head is DISEASE and and the ontotype of the second head is HUMAN then the relation is PATIENT&amp;quot; covering an instance as e.g. &amp;quot;iron deficiency in females&amp;quot;.</Paragraph>
      <Paragraph position="2"> The rule with the second highest coverage, and a fairly low precision of around 66%, is the rule: &amp;quot;if the ontotype of the second head is BODY PART then the relation type is LOCATIVE&amp;quot;. The rule covers instances as e.g. &amp;quot;...thrombosis in the heart&amp;quot; but also incorrectly classifies all instances as LOCATIVE where the relation type should be SOURCE. E.g. the sentence '...iron absorbtion from the intestine&amp;quot;, which is in fact a SOURCE relation, but is classified as LOCATIVE by the rule.</Paragraph>
      <Paragraph position="3"> One of the least surprising and most precise rules is: &amp;quot;if the ontotype of the second head is TIME then the relation type is TEMPORAL&amp;quot; covering an instance as e.g. &amp;quot;...diet for many months&amp;quot;. We would expect a similar rule to be produced, if we had performed the learning task on a general language corpus.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>