<?xml version="1.0" standalone="yes"?>
<Paper uid="W06-2708">
  <Title>Tools for hierarchical annotation of typed dialogue</Title>
  <Section position="3" start_page="0" end_page="57" type="metho">
    <SectionTitle>
2 Data Structures
</SectionTitle>
    <Paragraph position="0"> Our corpus has two sources of overlapping annotations: the turn structure of the corpus and situational factors annotation. The data are naturally split into turns whenever a participant presses their &amp;quot;submit&amp;quot; button. Timing information is associated with individual turns, representing the time when the entire message was sent to the other participant, rather than with individual words and sounds as it would be in spoken corpora.</Paragraph>
    <Paragraph position="1"> However, turns are too large to be used as units in the annotation for dialogue phenomena. For example, the single turn &amp;quot;Well done. Let's try a harder one.&amp;quot; consists of two utterances making different dialogue contributions: positive tutorial feedback for the previous student utterance and a statement of a new tutorial goal. Thus, turns must be segmented into smaller units which can serve as a basis for dialogue annotation. We call these utterances by analogy with spoken language, because they are often fragments such as &amp;quot;well done&amp;quot; rather than complete sentences.</Paragraph>
    <Paragraph position="2"> Thus, the corpus has two inherently overlapping layers: the turn segmentation layer, grouping utterances into turns, and the dialogue structure layer built up over individual utterances. The NXT toolkit supports such overlapping annotations, and we built two individual tools to support corpus annotation: an utterance segmentation tool and a tutorial annotation tool.</Paragraph>
    <Paragraph position="3"> Additionally, the corpus contains annotation done by the tutor herself at collection time which we call &amp;quot;situational factors&amp;quot;. The tutors were asked to submit a set of these factors after each turn describing the progress and state of the student, such as answer correctness, confidence and engagement. The factors were submitted separately from dialogue contributions and provide another layer of dialogue annotation which has to be coordinated with other annotations. The factors are typically related to the preceding student's utterance, but the link is implicit in the submission time.1 Currently we include the factors in the tool's transcript display based on the submission time, so they are displayed after the appropriate turn in the transcript allowing the annotators to visually synchronise them with the dialogue. We also provide an option to annotators for making them visible or not. In the future we plan to make factors a separate layer of the annotation linked by pointers with the preceding student and tutor turns.</Paragraph>
    <Paragraph position="4"> 1The factor interface was designed to be quick to use and minimally impact the dialogue flow, so the submission timings are generally reliable.</Paragraph>
  </Section>
  <Section position="4" start_page="57" end_page="57" type="metho">
    <SectionTitle>
3 Utterance Segmentation
</SectionTitle>
    <Paragraph position="0"> We process the raw data with an automatic segmenter/tokenizer which subdivides turns into individual utterances, and utterances into tokens, providing an initial segmentation for the annotation.</Paragraph>
    <Paragraph position="1"> However, perfect automatic segmentation is not possible, because punctuation is often either inconsistent or missing in typed dialogue and this task therefore requires human judgement. The output of our automatic segmentation algorithm was verified and corrected by a human annotator.</Paragraph>
    <Paragraph position="2"> A screen-shot of the interface we developed for segmentation verification is displayed in Figure 1.</Paragraph>
    <Paragraph position="3"> With the aid of this tool, it took 6 person-hours to check and correct the automatically segmented utterances for the 18 dialogues in our corpus.</Paragraph>
  </Section>
  <Section position="5" start_page="57" end_page="57" type="metho">
    <SectionTitle>
4 Tutorial Annotation
</SectionTitle>
    <Paragraph position="0"> To provide a detailed analysis of tutorial dialogue and remediation strategies, we employ a hierarchical annotation scheme which encodes the recursive dialogue structure. Each tutorial session consists of a sequence of tasks, which may be either teaching specific domain concepts or doing individual exercises. Each task's structure includes one or more of the following: giving definitions, formulating a question, obtaining the student answer and remediation by the tutor.</Paragraph>
    <Paragraph position="1"> Generally speaking, the structure of tutorial dialogue is governed by the task structure just as in task-oriented dialogue (Grosz and Sidner, 1986).</Paragraph>
    <Paragraph position="2"> However, the specific annotation structure differs depending on the tutoring method. In our basic electricity and electronics domain, a tutorial session consists of a set of &amp;quot;teach&amp;quot; segments, and within each segment a number of &amp;quot;task&amp;quot; segments. Task segments usually contain exercises in which the student is asked a question requiring a simple (one- or two-line) answer, which may be followed by a long remediation segment to address the conceptual problems revealed by the answer.</Paragraph>
    <Paragraph position="3"> In contrast, in our calculus domain the students have to do multi-step procedures to differentiate complex math expressions, but most of the remediations are very short, fixing the immediate problem and letting the student continue on with the procedure. Thus even though the dialogue is hierarchically structured in both cases, the annotation schemes differ depending on the domain. We developed a generic tool for annotating hierarchical dialogue structure which can be configured with the specific annotation scheme.</Paragraph>
    <Paragraph position="4"> The tool interface (Figure 2) consists of a transcript of a session and a linked tree representation. Individual utterances displayed in the transcript are leaves of the tree. It is not possible to display them as tree leaves directly as would be done in syntactic trees, because they are too large to fit in graphical tree display. Instead, a segment is highlighted in a transcript whenever it is selected in the tutorial structure, and a hotkey is provided to expand the tree to see all annotations of a particular utterance in the transcript.</Paragraph>
    <Paragraph position="5"> The hierarchical tree structure is supported by a schema which describes the annotations possible on each hierarchical tree level. Since the multi-layered annotation scheme is quite complex, the tool uses the annotation schema to limit the number of codes presented to the annotator to be only those consistent with the tree level. For example, in our basic electricity domain annotation described above, there are about 20 codes at different level, but an annotator will only have &amp;quot;teach&amp;quot; as an option for assigning a code to a top tree level, and only &amp;quot;task&amp;quot; and &amp;quot;test&amp;quot; (with appropriate subtypes) for assigning codes immediately below the teach level, based on the schema defined for the domain.</Paragraph>
  </Section>
  <Section position="6" start_page="57" end_page="57" type="metho">
    <SectionTitle>
5 Transcript Segmentation
</SectionTitle>
    <Paragraph position="0"> We had to conduct several simpler data analyses where the utterances in the transcript are segmented according to their purpose. For example, in tutorial differentiation the dialogue concentrates on 4 main purposes: general discussion, introducing problems, performing differentiation proper, or doing algebraic transformations to simplify the resulting expressions. In another analysis we needed to mark the segments where the student was making errors and the nature of those errors.</Paragraph>
    <Paragraph position="1"> We developed a generic annotation tool to support such segmentation annotation over the utterance layer. The tool is configured with the name of the segment tag and colours indicating different segment types. The annotator can enter a segment type, and use a freetext field for other information.</Paragraph>
    <Paragraph position="2"> A screenshot of the annotation tool with utterance purposes marked is given in Figure 3.</Paragraph>
  </Section>
  <Section position="7" start_page="57" end_page="59" type="metho">
    <SectionTitle>
6 Data Analysis
</SectionTitle>
    <Paragraph position="0"> The NITE query language (NQL) enables us to access the data as a directed acyclic graph to correlate simple annotations, such as finding out the  number of turns which contain only mathematical expressions but no words. We use the NITE query interface for simpler analysis tasks such as finding all instances of specific tags and tag combinations. However, we found the query language less useful for coordinating the situational factors annotated by tutors with other annotation. Each set of factors submitted is normally associated with the first student turn which precedes it, but the factors were not linked to student utterances explicitly.</Paragraph>
    <Paragraph position="1"> NQL does not have a &amp;quot;direct precedence&amp;quot; operator.2 Thus it is easier derive this information using the JAVA API. To make the data analysis simpler, we are planning to add a pointer layer, generated automatically based on timing information, which will use explicit pointers between the factor submissions and preceding tutor and student turns.</Paragraph>
  </Section>
  <Section position="8" start_page="59" end_page="59" type="metho">
    <SectionTitle>
7 Conclusions
</SectionTitle>
    <Paragraph position="0"> We presented a set of tools for hierarchically annotating dialogue structure, suitable for annotating typed dialogue. The turns in these dialogues are complex and overlap with dialogue structure, and our toolset supports segmenting turns into smaller 2It's possible to express the query in NQL using its precedence operator &amp;quot;lessmuch&amp;quot; as &amp;quot;($f factor) ($u utterance) (forall $u1 utterance) : (($f lessmuch $u) &amp;&amp; ($f lessmuch u1)) - (u lessmuch u1)&amp;quot;. However, this is very inefficient since it must check all utterance pairs in the corpus to determine direct precedence, especially if it needs to be included as part of a bigger query. utterance units and annotating hierarchical dialogue structure over the utterances, as well as providing simpler segmentation annotation.</Paragraph>
  </Section>
  <Section position="9" start_page="59" end_page="59" type="metho">
    <SectionTitle>
Acknowledgements
</SectionTitle>
    <Paragraph position="0"> This material is based upon work supported by a grant from The Office of Naval Research number N000149910165 and European Union 6th framework programme grant EC-FP6-2002-IST-</Paragraph>
  </Section>
class="xml-element"></Paper>