<?xml version="1.0" standalone="yes"?>
<Paper uid="W06-3813">
  <Title>Matching Syntactic-Semantic Graphs for Semantic Relation Assignment</Title>
  <Section position="7" start_page="85" end_page="86" type="evalu">
    <SectionTitle>
5 Experiments
</SectionTitle>
    <Paragraph position="0"> The system processes the 513 sentences interactively. It begins by running the DIPETT parser.</Paragraph>
    <Paragraph position="1"> Next, it extracts syntactic units (clauses, phrases, words) and pairs them up according to the information in the parse tree. Each unit is represented by its head word. Next, the system checks if the same pair of word lemmas has already been processed, to propose the same relation(s) to the user as options.</Paragraph>
    <Paragraph position="2"> If not, the system builds a graph centered on the head word, and proceeds with the matching on previously encountered instances, as described in section 4. When a set of candidates has been found, the system goes through a dialogue with the user.</Paragraph>
    <Paragraph position="3"> The system generated 2020 pairs from the 513 sentences. The experiment was run in 5 interactive sessions of approximately 3 hours each. The total net processing time was 6 hours, 42 minutes and 52 seconds4. While it would have been instructive to run the system several times with different users, it was not feasible. The experiment was run once, with two cooperating users. They were instructed on the set of semantic relations, and told how the system works. They discussed the semantic relation assignment and, once agreed, compared the system's suggestion with their decision.</Paragraph>
    <Paragraph position="4"> DIPETT did not produce a complete parse for all sentences. When a complete parse (correct or incorrect) was not possible, DIPETT produced fragmentary parses. The semantic analyser extracted units even from tree fragments, although sometimes the fragments were too small to accommodate any pairs.</Paragraph>
    <Paragraph position="5"> Of the 513 input sentences, 441 had a parse tree that allowed the system to extract pairs.</Paragraph>
    <Paragraph position="6">  Of 2020 pairs generated, the users discarded 545 in the dialogue step. An example of an erroneous pair comes from the sentence: Tiny clouds drift across like feathers on parade.</Paragraph>
    <Paragraph position="7"> The semantic analyser produced the pair (drift,parade), because of a parsing error: parade was parsed as a complement of drift, instead of a post-modi er for feathers. The correct pairing (feather,parade) is missing, because it cannot be inferred from the parse tree.</Paragraph>
    <Paragraph position="8"> Table 1 gives a summary of the processing statistics. We observe that graph-matching was used to process a clear majority of the total pairs extracted 63.25% (933/1475) , leaving the remaining 36.75% for the other two heuristics and for cases where no suggestion could be made. In 57.02% of the situations when graph-matching was used, the system's suggestion contained the correct answer (user's action was either accept or choose), and in 19.61% of the situations a single correct semantic relation was proposed (user action was accept).</Paragraph>
    <Paragraph position="9"> When the system presents multiple suggestions to the user, including the correct one, the average number of suggestions is 3.75. The small number of suggestions shows that the system does not simply add to the list relations that it has previously encountered, but it learns from past experience and graph-matching helps it make good selections. Figure 2 plots the difference between the number of examples for which the system gives the correct answer (possibly among other suggestions) and the number of examples when the user must supply the correct relation, from the rst example processed until the end of the experiment. We observe a steady increase in the number of correctly processed examples.</Paragraph>
    <Paragraph position="10"> Our system does not differentiate between syntactic levels, but based on the structures of the syntactic units in each pair we can decide which syntactic level it pertains to. For a more in-depth analysis, we have separated the results for each syntactic level,  correct relation and present them for comparison in Figure 3.</Paragraph>
    <Paragraph position="11"> We observe that the intra-clause level verbs and their arguments makes the best use of graphmatching, with the curve showing the cumulative number of situations in which the system makes correct predictions becoming steeper as more text is processed. At the same time, the curve that plots the cumulative number of cases in which the user has to supply a correct answer begins to level off. As expected, at the noun-phrase level where the syntactic structures are very simple, often consisting of only the noun and its modi er (without even a connective), the graph-matching algorithm does not help as much. At the inter-clause level the heuristic helps, as shown by the marginally higher curve for cumulative accept/choose user actions, compared to supply actions.</Paragraph>
  </Section>
class="xml-element"></Paper>