<?xml version="1.0" standalone="yes"?>
<Paper uid="W97-0310">
  <Title>Assigning Grammatical Relations with a Back-off Model</Title>
  <Section position="7" start_page="92" end_page="647" type="evalu">
    <SectionTitle>
4 Results
</SectionTitle>
    <Paragraph position="0"> The method described in the previous section was applied to a text corpus consisting of 5 months of the newspaper Frankfurter Allgemeine Zeitung with approximately 15 million word-like tokens. The learning procedure produced a total of 24,178 test tuples and 47,547 training triples.</Paragraph>
    <Section position="1" start_page="92" end_page="93" type="sub_section">
      <SectionTitle>
4.1 Learning procedure
</SectionTitle>
      <Paragraph position="0"> In order to evaluate the data used to train the model, 1000 training tuples were examined. Of these tuples, 127 were considered to be (partially) incorrect based on the judgments of a single judge given the original sentence. Errors in training and test data may stem from the morphology component, from the grammar specification, from the heuristic rule, or from actual errors in the text.</Paragraph>
      <Paragraph position="1">  The system works without subcategorization information; it suffices for a verb to occur with a possibly nominative and a possibly accusative NC for it to be considered training/test data. Lack of subcategorization leads to errors when verbs occurring with an (ambiguous) dative NC are mistaken for verbs which subcategorize for an accusative nominal phrase. For instance in (7) below, the verb gehSren ('to belong') takes, in one reading, a dative NP as its object and a nominative NP as its subject. Since the nominal constituent \[NC Bill\] is ambiguous with respect to case and possibly accusative, the erroneous tupie (Wagen, gehSren, Bill, 1) ('car, belong, Bill') is produced for this sentence.</Paragraph>
      <Paragraph position="2">  For instance in sentence (8), the verb trainieren ('to train') occurs with two NCs. Since the NC preceding the verb is unambiguously nominative and the one following the verb possibly accusative, the training tuple (Tennisspieler, trainieren, Jahr, 1) ('ten null nis player, train, year') is produced for this sentence, although the second NC is not an object of the verb. (8) Der Tennisspieler trainiert das ganze Jahr.</Paragraph>
      <Paragraph position="3"> the tennis player trains the whole year  In sentence (9) below, the word morgen ('tomorrow') is an adverb. However, its capitalized form may also be a noun, leading in this case to the erroneous training tuple (Morgen, trainieren, Tennisspieler, O) (since \[NO der Tennisspieler\] is unambiguously nominative).</Paragraph>
      <Paragraph position="4"> (9) Morgen trainiert der Tennisspieler.</Paragraph>
      <Paragraph position="5"> tomorrow trains the tennis player 'The tennis player will train tomorrow.'  In German, verb prefixes can be separated from the verb. When a finite (separable prefix) main verb occupies the second position in the clause, its prefix takes the last position in the clause core. For example in sentence (10) below, the prefix zur~ick of the verb zuriickweisen ('to reject') follows the object of the verb and a subordinate clause with a subjunctive main verb. This construct is not covered by the current version of the grammar. However, due to the grammar definition, and since weisen is also a verb (without a separable prefix) in German, \[c Er weist die Kritik der Prinzessin\] is still accepted as a valid clause, leading to the erroneous training tuple (er, weisen, Kritik, 1) ('he, point, criticism'). Such errors may be avoided with further development of the grammar.</Paragraph>
      <Paragraph position="6"> (10) Er weist die Kritik der Prinzessin, seine he rejects the criticism the princess his Ohren seien zu grofl, zurfick.</Paragraph>
      <Paragraph position="7"> ears are too big PRT 'He rejects the princess' criticism that his ears are too big.&amp;quot;  The system is not always able to determine constituent heads correctly. For instance in sentence (11), all words in the name Mexikanische Verband \]iir Menschenrechte are capitalized. Upon encountering the adjective Mexikanische, the system takes it to be a noun (nouns are capitalized in German), followed by the noun Verband &amp;quot;in apposition&amp;quot;. Sentence (11) is the source of the erroneous training tuple (Mexikanisch, beschuldigen, BehSrde, 1) ('Mexican, blame, public authorities').</Paragraph>
      <Paragraph position="8"> (11) Der Mexikanische Verband fiir Menschenthe Mexican Association for Human rechte beschuldigt die BehSrden.</Paragraph>
      <Paragraph position="9"> Rights blames the public authorities 'The Mexican Association for Human Rights blames the public authorities.'  The learning procedure has no access to multi-word lexical units. For instance in sentence (12), the first word in the expression Hand in Hand is considered the object of the verb, leading to the training tuple (Architekten, arbeiten, Hand, 1) ('architect, work, hand'). Given the information the system has access to, such errors cannot be avoided.</Paragraph>
      <Paragraph position="10"> (12) Alle Architekten sollen Hand in Hand arbeiten. all architects should hand in hand work 'All architects should work hand in hand.'  Not only spelling errors in the source text are the source of incorrect tuples. For instance in sentence (13), the verb suchen ('to seek') is erroneously in the third person plural. Since Reihe ('series') in German is a singular noun, and Kontakte ('contacts') plural, the actual object, but not the subject, agrees in number with the verb, so the incorrect tuple (Reihe, suchen, Kontakt, O) ('series, seek, contact') is obtained from this sentence.</Paragraph>
      <Paragraph position="11"> (13) *Eine Reihe von Staaten suchen gesch/iftliche a series from states seek business Kontakte zu der Region.</Paragraph>
      <Paragraph position="12"> contacts to the region '*A series of states seek contacts to the region.' Finally, a large number of errors, specially in test tuples, stems from the fact that soft constraints are used for words unknown to the morphology.</Paragraph>
    </Section>
    <Section position="2" start_page="93" end_page="647" type="sub_section">
      <SectionTitle>
4.2 Decision Algorithm
</SectionTitle>
      <Paragraph position="0"> In order to evaluate the accuracy of the decision algorithm, 1000 triples were selected from the set of test triples. Of these, 285 contained errors, based  on the judgements of a single judge given the original sentence 2. The results produced by the system for the remaining 715 tuples were compared to the judgements of a single judge given the original text. The system performed with an overall accuracy of 90.49%.</Paragraph>
      <Paragraph position="1"> A lower bound for the accuracy of the decision algorithm can be defined by considering the first noun in every test tuple to be the subject of the verb (by far the most common construct), yielding for these 715 tuples an accuracy of 87.83%.</Paragraph>
      <Paragraph position="2"> The above figure shows how many of the 715 evaluated test tuples were assigned subject/object based on the values Pn, and the accuracy of the system at each level.</Paragraph>
      <Paragraph position="3"> The accuracy for P2 and Ps exceeds 95%. However, their coverage is relatively low (28.81%). Since the procedure used to collect training data runs without supervision, increasing the size of the training set depends only on the availability of sample text and should be further pursued.</Paragraph>
      <Paragraph position="4"> One reason for the relatively low coverage is the fact that German compound nouns considerably increase the size of the sample space. For instance, the head of the nominal constituent \[NC Der Tennisspieler\] ('the tennis player') is considered by the system to be the compound noun Tennisspieler ('tennis player'), instead of its head noun Spieler ('player'). Consistently considering the head of putative compound nouns to be the head of nominal constituents may in some cases lead to awkward results. However, reducing the size of the sample space by morphological processing of compound nouns should be considered in order to increase coverage. null  Following are examples of test tuples for which a decision was made based on values of P2. All sentences below stem from the corpus.</Paragraph>
      <Paragraph position="5"> Sentence (14) was the source for the test tuple (Ausstellung, zeigen, Spektrum) ('exhibition, show, 2The higher error rate for test tuples is due to the soft constraints used for words unknown to the morphology. spectrum'). This tuple was correctly disambiguated with P2 = 0.87, with, among others, the training tuples (Ausstellung, zeigen, Bild, 1) ('exhibition, show, painting'), (Ausstellung, zeigen, Beispiel, 1) ('exhibition, show, example'), and (Ausstellung, zeigen, Querschnitt, 1) ('exhibition, show, crosssection') obtained with the Agreement (sentences (15) and (16)) and Case Rules (sentence (17)), respectively. null (14) Die Ausstellung zeigt das Spektrum jfidischer the exhibition shows the spectrum jewish Buchkunst yon den AnFdngen \[...\] book art from the beginnings 'The exhibition shows the spectrum of jewish book art from the beginnings \[...\].' (15) die letzte Ausstellung vor der Sommerpause the last exhibition before the summer pause zeigt Bilder und Zeichnungen von Petra shows paintings und drawings from Petra Trenkel zum Thema &amp;quot;Dorf&amp;quot;.</Paragraph>
      <Paragraph position="6"> Trenkel to the subject village 'The last exhibition before the summer pause shows paintings and drawings by Petra Trenkel on the subject &amp;quot;village&amp;quot;.' (16) Die Ausstellung im Museum fiir Kunstthe exhibition in the museum for arts and handwerk zeigt Beispiele seiner vielf~iltigen crafts shows examples his manifold Objekt-Typen \[...\] object types 'The exhibition in the museum for arts and crafts shows examples of his manifold object types \[...\]' (17) Eine vom franzSsischen Kulturinstitut a from the French culture institute mit Unterstiitzung des BSrsenvereins with support the BSrsenverein in der Zentralen Kinder- und Jugendbibliothek in the central children and youth library  im Biirgerhaus Bornheim in the community center Bornheim eingerichtete Ausstellung zeigt organized exhibition shows einen interessanten Querschnitt.</Paragraph>
      <Paragraph position="7"> an interesting cross-section 'A exhibition in the central children's and youth library in the community center Bornheim, organized by the French culture institute with support of the BSrsenverein, shows an interesting cross-section.' Sentence (18) below was the source for the test tuple (Altersgrenze, nennen, Gesetz) ('age limit, mention, law'). The system incorrectly considered the noun Altersgrenze to be the subject of the verb.</Paragraph>
      <Paragraph position="8"> (18) Eine Altersgrenze nennt das Gesetz nicht. an age limit mentions the law not 'The law does not mention an age limit.' There were no training tuples in which the compound noun Altersgrenze occurred as the subject/object of the verb. However, the noun Gesetz occurred more frequently as the object of the verb nennen than as its subject, leading to the erroneous decision.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>