<?xml version="1.0" standalone="yes"?>
<Paper uid="W06-1803">
  <Title>Interpretation and Generation in a Knowledge-Based Tutorial System</Title>
  <Section position="4" start_page="0" end_page="0" type="metho">
    <SectionTitle>
2 Motivation
</SectionTitle>
    <Paragraph position="0"> A good teaching method for basic electricity and electronics is eliciting cognitive dissonance (Schaffer and McDermott, 1992; Arnold and Millar, 1987) which we are implementing as a &amp;quot;predict-verify-evaluate&amp;quot; (PVE) cycle. The students are asked to make predictions about the behavior of a schematic circuit and then build it in a simulation environment. If the observed results do not match their predictions, a discussion ensues, where the computer tutor helps a student learn the relevant concepts. The PVE exercises are complemented with exercises asking the students to identify properties of circuits in diagrams and to interpret a circuit's behavior.</Paragraph>
    <Paragraph position="1"> Thus, the system has to answer questions about circuits which students build and manipulate dynamically in a simulation environment, and produce explanations and feedback tailored to that individual context. This relies on the following system capabilities: * Understanding and giving explanations.</Paragraph>
    <Paragraph position="2"> Since the system relies on inducing cognitive dissonance, it should be able to explain to the student why their prediction for a specific circuit was incorrect, and also verify explanations given by a student.</Paragraph>
    <Paragraph position="3"> * Unrestricted language input with reference resolution. Similar to other conceptual domains (VanLehn et al., 2002) the language observed in corpus studies is varied and syntactically complex. Additionally, in our domain students refer to items on screen, e.g.</Paragraph>
    <Paragraph position="4"> &amp;quot;the lightbulb in 5&amp;quot;, which requires the system to make the connection between the language descriptions and the actual objects in the environment.</Paragraph>
    <Paragraph position="5"> * Tailored generation. The level of detail in the explanations offered should be sensitive to student knowledge of the domain. Tutorial utterances should be natural and use correct terminology even if a student doesn't.</Paragraph>
    <Paragraph position="6"> To support answering questions and giving explanations, we chose the KM knowledge representation environment (Clark and Porter, 1999) as a basis for our implementation. KM is a descriptionlogic based language which has been used to represent facts and rules in a HALO system for AP chemistry tests (Barker et al., 2004). It supports the generation of explanations and obtained the highest explanation scores in an independent evaluation based on an AP chemistry exam (Friedland et al., 2004). Thus it is a good choice to provide reasoning support for explanations and answering novel questions in a tutorial system. However, KM has not been used previously in connection with natural language input for question answering, and we discuss how the limitations of KM representations affect the interpretation process in Section 4. We use a deep domain-independent parser and grammar to support language interpretation, and a deep generator to provide natural sounding and context-dependent text. Both deep parsing and generation provide the context adaptivity we need, but they are time-consuming to build for a specific domain. Now that a number of deep domain-independent parsing and generation systems are available in the community, our research goal is to investigate the issues in integrating them with the knowledge representation for question answering to support the requirements of a tutorial dialogue system. We focus on context-dependent explanation understanding and generation as a primary tutoring task in our domain. Section 3 discusses our representations, Section 4 presents the issues arising in knowledge representation to support interpretation, and Section 5 discusses the requirements for appropriate explanation generation and how it can be integrated into the system.</Paragraph>
  </Section>
  <Section position="5" start_page="0" end_page="0" type="metho">
    <SectionTitle>
3 Representations
</SectionTitle>
    <Paragraph position="0"> From the point of view of tutoring, the most important requirement on the knowledge representation is that system reasoning should closely match human reasoning, so that it can be explained to students in meaningful terms. Thus, for example, a numerical circuit simulator is well suited for dynamically displaying circuit behaviors, but not for conceptually tutoring basic circuits, because it 5 KRAQ06 hides physics principles behind complex mathematical equations that are not suitable for learners. To design our knowledge representation we started with a set of lessons for our domain designed by psychologists experienced in designing training courses for physics and simulated environments. The lessons were used in a data collection environment with experienced tutors conducting tutoring sessions over a text chat interface. Each student and tutor were required to go through the materials presented as a set of slides and solve pre-defined exercises, but the students asked questions to get help with problem-solving. The tutor had complete freedom to choose how to answer student questions and how to remediate when students made mistakes. We are using this data set to study the types of errors that students make as well as the language used by both students and tutors. The latter serves as a guide to developing our interpretation and generation components.</Paragraph>
    <Paragraph position="1"> In addition to the set of materials, the course designers provided a &amp;quot;glossary&amp;quot; of concepts and facts that students need to learn and use in explanations, containing approximately 200 concepts and rules in a form which should be used in model explanations. We then developed our knowledge representation so that concepts listed in the glossary were represented as KM concepts, and facts are represented as rules for computing slots.</Paragraph>
    <Paragraph position="2"> An example KM representation for our domain is shown in Figure 1. It represents the fact that a lightbulb will be on if it is in a complete path (i.e. a closed path containing a battery). The explanation is generated using the comment structure [lightbulbstate] shown in Figure 2 (slightly simplified for readability). Explanations are generated separately from reasoning in the KM system because reasoning in general contains too many low-level details. For example, our rule for computing a lightbulb state includes two facts: that the lightbulb has to be in a complete path with a battery, and that a lightbulb is always in one state only (i.e.</Paragraph>
    <Paragraph position="3"> it cannot be broken and on at the same time). The latter is required for proof completeness, but is too trivial to be mentioned to students. KM therefore requires knowledge engineers to explicitly designate the facts to be used in explanations.</Paragraph>
    <Paragraph position="4"> This representation allows KM to generate detailed explanations by using a template string and then explaining the supporting facts. An example of a full explanation, together with the adjustments needed to use it in dialogue rather than as a complete answer, is given in Section 5.</Paragraph>
    <Paragraph position="5"> Currently, KM only supports generating explanations, but not verifying them. The explanation mechanism produces explanations as text directly from the knowledge representation, as shown in Figure 2(a). This generation method is not well suited for a tutorial dialogue system, because it does not take context into account, as discussed in Section 5. Therefore, we are designing a structured representation for explanations to be produced by the KM explanation mechanism instead of using English sentences, shown in Figure 2(b).</Paragraph>
    <Paragraph position="6"> This will allow us to generate more flexible explanations (Section 5) and also to interpret student explanations (Section 4).</Paragraph>
  </Section>
  <Section position="6" start_page="0" end_page="0" type="metho">
    <SectionTitle>
4 Interpretation
</SectionTitle>
    <Paragraph position="0"> The interpretation process consists of parsing, reference resolution, dialogue act recognition and diagnosing student answers. We discuss reference resolution and diagnosis here as these are the two steps impacted by the knowledge representation issues. As a basic example we will use the student answer from the following pair:1 Problem: For each circuit, which lightbulbs will be lit? Explain.</Paragraph>
    <Paragraph position="1"> Student: the bulbs in 1 and 3 are lit because they are in a closed path with a battery To respond to this answer properly, the system must complete at least the following tasks. First, it must resolve &amp;quot;the bulbs in 1 and 3&amp;quot; to corresponding object IDs in the knowledge base, for example, LB-13-1-1 and LB-13-3-1. Then, it must verify that the student statement is factually correct. This includes verifying that the lightbulbs in 1 and 3 will be lit, and that each of them is in a closed path with a battery. Finally, it must verify that the student explanation is correct. This is separate from verifying factual correctness. For example, a statement &amp;quot;because they are in a closed path&amp;quot; is true for both of those lightbulbs, but it is not a complete explanation, because a lightbulb may be in a closed path which does not contain a battery, where it won't be lit.</Paragraph>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
4.1 Interpreting Factual Statements
</SectionTitle>
      <Paragraph position="0"> We use the TRIPS dialogue parser (Dzikovska, 2004) for interpretation. The TRIPS parser provides a two-layer architecture where the utterance meaning is represented using a domain-independent semantic ontology and syntax. The domain-independent representation is used for discourse processing tasks such as reference resolution, but it is connected to the domain-specific knowledge representation by mapping between the domain-independent and domain-specific ontologies (Dzikovska et al., 2003; Dzikovska, 2004). This architecture allows us to separate linguistic and domain-specific knowledge and easily specialize to new domains.</Paragraph>
      <Paragraph position="1"> When applied in our domain, the TRIPS interpretation architecture was helpful in getting the interpretation started quickly, because we only needed to extend the lexicon with specific terms related to basic electricity and electronics (e.g., &amp;quot;multimeter&amp;quot;), while other lexical items and syntactic constructions were provided in the domain-independent part.</Paragraph>
      <Paragraph position="2"> The reference resolution module operates on TRIPS domain-independent representations sending queries to KM as necessary, because the TRIPS representations offer linguistic features to guide reference resolution not available in the representations used for reasoning. We use a recursive reference resolution algorithm similar to Byron (2002) which first resolves &amp;quot;1 and 3&amp;quot; are resolved as names for Circuit-13-1 and Circuit-133,2 and then queries KM to find all lightbulbs in those circuits. Dialogue context is used to interpret the reference resolution results. In this case, the context does not matter because the question sets up all lightbulbs on screen as contextually relevant. But if the student had said &amp;quot;the ones in 1 and 3&amp;quot;, the query would be for all components in circuits 1 and 3, and then our algorithm will filter the query results based on the question context to retain only lightbulbs.</Paragraph>
      <Paragraph position="3"> Once the references are resolved, the whole sentence is converted to a KM statement which represents the student utterance, in our case (the state of LB13-1-1) = *On-Usage-State, where LB13-1-1 is the lightbulb obtained by reference resolution. This statement is sent to the KM system, which verifies that it is correct. This procedure allows us to use dialogue context in understanding, and also to check correctness of answers easily, even if they are phrased in an unanticipated way.</Paragraph>
      <Paragraph position="4"> However, even with the layer of separation of linguistic and domain knowledge provided by the TRIPS architecture, we found that the need to support interpretation in a compositional way influences the interaction with knowledge representation. There are many ways to express the same query to KM, which differ in efficiency. Two ex- null &amp;quot;1&amp;quot; refers to terminals or other components rather than whole circuits, and therefore there is no 1-to-1 correspondence between names and objects in the environment.</Paragraph>
      <Paragraph position="5"> 7 KRAQ06 (a) (allof ?x in (the all-instances of LightBulb) where ((the components of Circuit-13-1) include ?x)) (allof ?x (LightBulb ?x) and (components Circuit-13-1 ?x)) (b) (allof ?comp in (the components of Circuit-13-1) where (?comp isa LightBulb))  equivalent except for the order of conjuncts, they are expressed in a very different way in the KM syntax. Version (b) is more efficient to ask, because it retrieves the components of circuit 1 first, a smaller set than the set of all lightbulbs.</Paragraph>
      <Paragraph position="6"> This asymmetry presents a challenge to both language interpretation and knowledge engineering. Existing reference resolution algorithms (Byron, 2002; Bos, 2004) expect the queries for &amp;quot;the lightbulb&amp;quot; and &amp;quot;the lightbulb in 1&amp;quot; to be strictly compositional in the sense that the phrase &amp;quot;the lightbulb&amp;quot; will be represented identically in both cases, and &amp;quot;in 1&amp;quot; is represented as an additional constraint on the lightbulbs. This corresponds to the query variant (a) in the system. Otherwise a large amount of query-specific transformations may be required to produce queries for complex noun phrase descriptions, diminishing the scalability of the approach.</Paragraph>
      <Paragraph position="7"> We had to spend a significant portion of time in the project developing an efficient and compositional knowledge representation. Our current solution is to prefer compositionality over efficiency, even though it impacts performance in some cases, but we are working on a more general solution. Instead of converting directly to KM from domain-independent language representations, we will convert all queries in a FOL-like syntax shown in Figure 3 which uses concepts from the KM representation, but where all conjuncts are treated identically in the syntax. The problem of converting this representation to the optimal KM form can then be seen as an instance of query optimization. For example, we can re-order the conjuncts putting the relations which include an instance constant (e.g., (the components of Circuit-13-1)) first in the query, because they are more likely to limit the search to small sets of objects. This representation can be easily converted in the KM syntax, and is also useful for explanation understanding and generation as discussed below.</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
4.2 Explanation Understanding
</SectionTitle>
      <Paragraph position="0"> While KM has facilities for generating explanations, it does not have support for reading in a student explanation and verifying it. We devised a method to support this functionality with the aid of KM explanation generation mechanism. Any time a student offers an explanation, the KM reasoner will be called to generate its own explanation for the same fact, in the structured format shown in Figure 2(b). Then the student explanation (converted into the same intermediate syntax) can be matched against the KM-generated explanation to verify that it is complete, or else that certain parts are missing.</Paragraph>
      <Paragraph position="1"> In our example, the student explanation &amp;quot;because they are in a closed path with a battery&amp;quot; will be represented as (?pa instance-of Path) (?pa is-closed t) (?b instance-of Battery) (?pa contains ?b) (?pa contains LB-13-1-1).3 This explanation does not directly match into the explanation structure from Figure 2(b), because it uses the more specific term &amp;quot;in closed path with a battery&amp;quot; rather than the more general term &amp;quot;in complete path&amp;quot; (represented by the powered-by slot). However, as part of generating the explanation, an explanation structure for the powered-by will be generated, and it will include the facts (?pa is-closed t) (?pa contains ?b). This will match the student explanation. It will be up to the tutorial module to decide whether to accept the explanation &amp;quot;as is&amp;quot;, or lead the student to use the more precise terminology, as discussed in Section 5.</Paragraph>
      <Paragraph position="2"> This method can address student explanations as long as they correspond to parts of typical explanations, and identify missing parts. The biggest open problem we face is equivalent inferences.</Paragraph>
      <Paragraph position="3"> For example, a student may say &amp;quot;A lightbulb is not on&amp;quot; instead of &amp;quot;a lightbulb is off&amp;quot;. KM reasoning handles those differences when verifying factual correctness, but KM does not support similar reasoning for matching explanations (which 3Here &amp;quot;they&amp;quot; would be resolved first to a set of lightbulbs, and each instance will be treated separately to verify that the explanation applies.</Paragraph>
      <Paragraph position="4"> 8 KRAQ06 would correspond to verifying full proofs rather than individual facts). We are considering bringing a theorem prover to reason over intermediate representations together with KM axioms to help interpret explanations, as done in (Makatchev et al., 2004; Bos, 2005).</Paragraph>
    </Section>
  </Section>
  <Section position="7" start_page="0" end_page="0" type="metho">
    <SectionTitle>
5 Generation
</SectionTitle>
    <Paragraph position="0"> The task of the utterance generation component is to produce tutorial dialogue, such as asking new questions of the student, conveying the correctness of their answers, and giving explanations. Explanations may be given in response to a student's direct 'why' question or when a student has erred and the pedagogical reasoner has decided that an explanation is the best remediation strategy. In each case, the utterance generator must not only provide a correct, thorough and coherent explanation, but must tailor it so that the student doesn't receive too much or too little information. To be tailorable, explanations must be derived from the represented domain knowledge and from what the tutoring system knows about the student (e.g., their recent performance).</Paragraph>
    <Paragraph position="1"> Directly producing explanations by appending together pieces of hand-written strings as in Figure 2(a) usually results in long explanations that contain little detail of interest to the student. Figure 4 contains one such example explanation generated by the KM system in our domain and derived from a query based on the production rule in  ing an exam question, as intended in the KM system, but it is not necessarily helpful in dialogue. As an example, suppose the student had incorrectly answered the question in Section 4, and the tutoring system decides to correctly explain why the lightbulbs are lit. Usually, a full explanation is not necessary in these cases. In the case where a student gave an incomplete explanation, namely leaving out the necessary mention of the battery, a simple response of the form &amp;quot;Yes, but don't forget the battery&amp;quot; will be infinitely more helpful than the full explanation. If the student's explanation is completely correct, but they have failed to notice a change in the environment, the more appropriate explanation is &amp;quot;The lightbulb is in a closed path, as well as the battery, but the battery is not operational&amp;quot;. Furthermore, if a student has shown that they are knowledgeable about certain fundamental facts, such as what states a lightbulb may be in, statements like &amp;quot;A lightbulb can be on, off or broken&amp;quot; should be removed.</Paragraph>
    <Paragraph position="2"> Adding this reasoning directly to the knowledge base would make it unwieldy and unmodifiable, and the string-based generation in KM comments does not allow for adapting explanations based on external knowledge such as a student model. To adapt the KM explanation mechanism to support such context-dependent generation, instead of creating explanations via template strings, we have devised the representation presented in Figure 2(b) that is based on semantics and allows us to modify an explanation after it has been produced by the KM reasoning process but before it has been converted into a string representation.</Paragraph>
    <Paragraph position="3"> Based on this semantic representation, explanation content can be selected more appropriately. If the interpreter discussed in Section 4.2 determines that parts of the explanation from the :requires field are missing, the generation can focus only on that part of the explanation. The requirements list would also be used to determine if the student is not aware of environment properties, such as that a battery is damaged. Finally, the facts known to the student can be removed if the corresponding semantic forms were used in previous explanations.</Paragraph>
    <Paragraph position="4"> In addition to selecting the explanation content properly, it is important that the responses given to the student sound fluid and are easy to understand.</Paragraph>
    <Paragraph position="5"> In dialogue, in particular, it is important that pronouns can be generated based on references important for the student, and avoid repetitiveness in syntax. Knowledge of linguistic features such as number and gender, and also knowledge of what was previously mentioned in the discourse, is necessary to support such natural text generation.</Paragraph>
    <Paragraph position="6"> Deep generation utilizes this represented knowledge along with grammatical and lexical knowledge of a language, rather than hand-written strings, to produce utterances. Our current implementation uses a custom utterance generation component and the STORYBOOK (Callaway and Lester, 2002) deep text generator modified to work in a dialogue context. Once the explanation content is selected, it is passed to the STORYBOOK system to produce the actual utterance text.</Paragraph>
  </Section>
class="xml-element"></Paper>