<?xml version="1.0" standalone="yes"?>
<Paper uid="E99-1027">
  <Title>Specifying a shallow grammatical representation</Title>
  <Section position="4" start_page="205" end_page="206" type="metho">
    <SectionTitle>
3 Preparations for the experiment
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="205" end_page="206" type="sub_section">
      <SectionTitle>
3.1 Experimental setting
</SectionTitle>
      <Paragraph position="0"> The experiment was conducted as follows.</Paragraph>
      <Paragraph position="1">  the analysis of unrecognlsed words, we used a rule-based heuristic component that assigns morphological analyses, one or more, to each word not represented in the lexicon of the system. Of the analysed text, two identical versions were made, one for each linguist.</Paragraph>
      <Paragraph position="2"> 2. Two linguists trained to disambiguate the ENGCG morphological representation (see the subsection on training below) independently marked the correct alternative analyses in the ambiguous input, using mainly structural, but in some structurally unresolvable cases also higher-level, information. The corpora consisted of continuous text rather than isolated sentences; this made the use of textual knowledge possible in the selection of the correct alternative. In the rare cases where two analyses were regarded as equally legitimate, both could be marked. The judges were encouraged to consult the documentation of the grammatical representation. In addition, both linguists were provided with a checking program to be used after the text was analysed. The program identifies words left without an analysis, in which case the linguist was to provide the m~.~sing analysis.</Paragraph>
      <Paragraph position="3"> 3. These analysed versions of the same text were compared to each other using the Unix sdiff program. For each corpus version, words with a different analysis were marked with a &amp;quot;RECONSIDER&amp;quot; symbol. The &amp;quot;RECONSIDER&amp;quot; symbol was also added to a number of other ambiguous words in the corpus. These additional words were marked in order to 'force' each linguist to think independently about the correct analysis, i.e. to prevent the emergence of the situation where one linguist considers the other to be always right (or wrong) and so 'reconsiders' only in terms of the existing analysis. The linguists were told that some of the words marked with the &amp;quot;RECONSIDER&amp;quot; symbol were analysed differently by them.</Paragraph>
      <Paragraph position="4"> 4. Statistics were generated about the number of differing analyses (number of &amp;quot;RECONSIDER&amp;quot; symbols) in the corpus versions (&amp;quot;diffl&amp;quot; in the following table).</Paragraph>
      <Paragraph position="5"> 5. The reanalysed versions were automatically compared to each other. To words with a different analysis, a &amp;quot;NEGOTIATE&amp;quot; symbol was added.</Paragraph>
      <Paragraph position="6">  were jointly examined by the linguists in order to see whether they were due to (i) inattention on the part of one linguist (as a result of which a correct unique analysis was jointly agreed upon), (ii) joint uncertainty about the correct analysis (both linguists feel unsure about the correct analysis), or (iii) conflicting opinions about the correct analysis (both linguists have a strong but different opinion about the correct analysis).</Paragraph>
      <Paragraph position="7"> 8. Statistics were generated about the number of conflicting opinions (&amp;quot;dill3&amp;quot; below) and joint uncertainty (&amp;quot;unsure&amp;quot; below).</Paragraph>
      <Paragraph position="8"> This routine was successively applied to each text.</Paragraph>
    </Section>
    <Section position="2" start_page="206" end_page="206" type="sub_section">
      <SectionTitle>
3.2 Training
</SectionTitle>
      <Paragraph position="0"> Two people were hired for the experiment. One had recently completed a Master's degree from English Philology. The other was an advanced undergraduate student majoring in English Philology. Neither of them were familiar with the ENGCG tagger.</Paragraph>
      <Paragraph position="1"> All available documentation about the linguistic representation used by ENGCG was made available to them. The chief source was chapters 3-6 in Karlsson et al. (eds., 1995). Because the linguistic solutions in ENGCG are largely based on the comprehensive descriptive grammar by Quirk et al. (1985), also that work was made available to them, as well as a number of modern English dictionaries.</Paragraph>
      <Paragraph position="2"> The training was based on the disambiguation of ten smallish text extracts. Each of the extracts was first analysed by the ENGCG morphological analyser, and then each trainee was to independently perform Step 3 (see the previous subsection) on it. The disambiguated text was then automatically compared to another version of the same extract that was disambiguated by an expert on ENGCG. The ENGCG expert then discussed the analytic differences with the trainee who had also disambiguated the text and explained why the expert's analysis was correct (almost always by identifying a relevant section in the available ENGCG documentation; in very rare cases where the documentation was underspecific, new documentation was created for future use in the experiments). null After analysis and subsequent consultation with the ENGCG expert, the trainee processed the fob lowing sample.</Paragraph>
      <Paragraph position="3"> The training lasted about 30 hours. It was concluded by familiarising the linguists with the routine used in the experiment.</Paragraph>
    </Section>
    <Section position="3" start_page="206" end_page="206" type="sub_section">
      <SectionTitle>
3.3 Test corpus
</SectionTitle>
      <Paragraph position="0"> Four texts were used in the experiment, totailing 55724 words and 102527 morphological analyses (an average of 1.84 analyses per word). One was an article about Japanese culture ('Pop'); one concerned patents ('Pat'); one contained excerpts from the law of California; one was a medical text ('Med'). None of them had been used in the development of the ENGCG grammatical representation or other parts of the system. By mid-June 1999, a sample of this data will be available for inspection at http://www.ling.helsinki.fi/ voutilai/eac199data.html. null</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>