<?xml version="1.0" standalone="yes"?>
<Paper uid="W00-1402">
  <Title>A Task-based Framework to Evaluate Evaluative Arguments</Title>
  <Section position="1" start_page="0" end_page="0" type="abstr">
    <SectionTitle>
Abstract
</SectionTitle>
    <Paragraph position="0"> We present an evaluation framework in which the effectiveness of evaluative arguments can be measured with real users.</Paragraph>
    <Paragraph position="1"> The framework is based on the task-efficacy evaluation method. An evaluative argument is presented in the context of a decision task and measures related to its effectiveness are assessed. Within this framework, we are currently running a formal experiment to verify whether argument effectiveness can be increased by tailoring the argument to the user and by varying the degree of argument conciseness.</Paragraph>
    <Paragraph position="2"> Introduction Empirical methods are fundamental in any scientific endeavour to assess progress and to stimulate new research questions. As the field of NLG matures, we are witnessing a growing interest in studying empirical methods to evaluate computational models of discourse generation (Dale, Eugenio et al. 1998).</Paragraph>
    <Paragraph position="3"> However, with the exception of (Chu-Carroll and Carberry 1998), little attention as been paid to the evaluation of systems generating evaluative arguments, communicative acts that attempt to affect the addressee's attitudes (i.e. evaluative tendencies typically phrased in terms of like and dislike or favor and disfavor).</Paragraph>
    <Paragraph position="4"> The ability to generate evaluative arguments is critical in an increasing number of online systems that serve as .personal :assistants, advisors, or sales asslstants ~. For instance, a travel assistant may need to compare two vacation packages and argue that its current user should like one more than the other.</Paragraph>
    <Paragraph position="5"> i See for instance www.activebuyersguide.com In this paper, we present an evaluation framework in which the effectiveness of evaluative arguments can be measured with real users. The measures of argument effectiveness used in our framework are based on principles developed in social psychology to study persuasion (Miller and Levine 1996). We are currently applying the framework to evaluate arguments generated by an argument generator we have developed (Carenini 2000). To facilitate the evaluation of specific aspects of the generation process, the argument generator has been designed so that its functional components can be easily turned-offor changed.</Paragraph>
    <Paragraph position="6"> In the remainder of the paper, we first describe our argument generator. Then, we summarize literature on persuasion from social psychology. Next, we discuss previous work on evaluating NLG models. Finally, we describe our evaluation framework and the design of an experiment we are currently running.</Paragraph>
  </Section>
class="xml-element"></Paper>