<?xml version="1.0" standalone="yes"?>
<Paper uid="W97-1502">
  <Title>The TreeBanker: a Tool for Supervised Training of Parsed Corpora</Title>
  <Section position="7" start_page="13" end_page="14" type="evalu">
    <SectionTitle>
5 Evaluation and Conclusions
</SectionTitle>
    <Paragraph position="0"> Using the TreeBanker, it is possible for a linguistically aware non-expert to judge around 40 sentences per hour after a few days practice. When the user becomes still more practised, as will be the case if he judges a corpus of thousands of sentences, this figure rises to around 170 sentences per hour in the case of our most experienced user. Thus it is reasonable to expect a corpus of 20,000 sentences to be judged in around three person weeks. A much smaller amount of time needs to be spent by experts in making judgments he felt unable to make (perhaps for one per cent of sentences once the user has got used to the system) and in checking the user's work (the TreeBanker includes a facility for picking out sentences where errors are mostly likely to have been made, by searching for discriminants with unusual values). From these figures it would seem that the TreeBanker provides a much quicker and less skill-intensive way to arrive at a disambiguated set of analyses for a corpus than the manual annotation scheme involved in creating the Penn Treebank; however, the TreeBanker method depends on the prior existence of a grammar for the domain in question, which is of course a non-trivial requirement. null Engelson and Dagan (1996) present a scheme for selecting corpus sentences whose judging is likely to provide useful new information, rather than those that merely repeat old patterns. The TreeBanker offers a related facility whereby judgments on one sentence may be propagated to others having the same sequence of parts of speech. This can be combined with the use of representative corpora in the CLE (Rayner, Bouillon and Carter, 1995) to allow only one representative of a particular pattern, out of perhaps dozens in the corpus as a whole, to be inspected. This already significantly reduces the number of sentences needing to be judged, and hence the time required, and we expect further reductions as Engelson's and Dagan's ideas are applied at a finer level.</Paragraph>
    <Paragraph position="1"> In the current implementation, the TreeBanker only makes use of context.independent properties: those derived from analyses of an utterance that are constructed without any reference to the context of use. But utterance disambiguation in general requires the use of information from the context. The context can influence choices of word sense, syntactic structure and, most obviously, anaphoric reference (see e.g. Carter, 1987, for an overview), so it might seem that a disambiguation component trained only on context-independent properties cannot give adequate performance.</Paragraph>
    <Paragraph position="2"> However, for QLFs for the ATIS domain, and presumably for others of similar complexity, this is not in practice a problem. As explained earlier, anaphors are left unresolved at the stage of analysis and disambiguation we are discussing here; and contextual factors for sense and structural ambiguity resolution are virtually always &amp;quot;frozen&amp;quot; by the constraints imposed by the domain. For example, although there are certainly contexts in which &amp;quot;Tell me flights to Atlanta on Wednesday&amp;quot; could mean &amp;quot;Wait until Wednesday, and then tell me flights to Atlanta&amp;quot;, in the ATIS domain this reading is impossible and so &amp;quot;on Wednesday&amp;quot; must attach to  &amp;quot;flights&amp;quot;. For a wider domain such as NAB, one could perhaps attack the context problem either by an initial phase of topic-spotting (using a different set of discriminant scores for each topic category), or by including some discriminants for features of the context itself among these to which training was applied.</Paragraph>
  </Section>
class="xml-element"></Paper>