<?xml version="1.0" standalone="yes"?>
<Paper uid="C04-1196">
  <Title>Understanding Students' Explanations in Geometry Tutoring</Title>
  <Section position="6" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
4 Performance Evaluation
</SectionTitle>
    <Paragraph position="0"> As a measure of the system's ability to understand students' explanations we evaluated the accuracy of the classification of these sentences with respect to the hierarchy of explanation classes. The evaluation used a set of 700 sentences representing actual explanations provided by high school students during an experimental study in 2003. The classification task consists in associating each sentence with one or more of 200 fine-grained categories, a difficult task even for humans. We used the kappa statistic (Cohen 1960) to measure the inter-rater reliability between the system and two human raters. We used three different measures.</Paragraph>
    <Paragraph position="1"> First, a &amp;quot;set equality&amp;quot; measure, where two sets of classes match only if they are identical. Second, an &amp;quot;overlap&amp;quot; measure, where two sets are considered to partially match if they share some subset. And third, a &amp;quot;weighted overlap&amp;quot;, which takes into account the relative semantic distance between different classes in assessing the match between two sets of categories. The results in Table 1 show the system to work reasonably well, although not at human level.</Paragraph>
    <Paragraph position="2">  human raters.</Paragraph>
    <Paragraph position="3"> Regarding the hypothesis question of whether replacing menu-based justifications with natural language justifications helps students have a better understanding of geometry, we do not have a definitive answer yet. Some experimental results based on the same study seem to show (Aleven et al. 2004) that while students' ability to express their knowledge was improved considerably, students' performance on actual problem solving was not affected significantly.</Paragraph>
    <Paragraph position="4"> There are a number of possible causes for that, so further studies are needed.</Paragraph>
  </Section>
class="xml-element"></Paper>