<?xml version="1.0" standalone="yes"?>
<Paper uid="W03-0206">
  <Title>Transforming Grammar Checking Technology into a Learning Environment for Second Language Writing</Title>
  <Section position="4" start_page="0" end_page="0" type="metho">
    <SectionTitle>
3 Naturalistic Studies on the Use of a
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
Swedish Language Tool
</SectionTitle>
      <Paragraph position="0"> We conducted two studies, first a pilot study and then a more systematic study called study 1. In the pilot study conducted, we aimed at developing and testing a methodology for data collection in the context of second language learners using Granska. We were also interested in the impact of the grammar checker on the writers' texts, and how the users have adapted the tool to their writing purposes. In the second study we were interested in how the methodology developed in the pilot study would work in an educational setting. What we found from the empirical studies resulted in a shift of focus. In fact, we became more interested in developing new functionality for a whole second language-learning environment allowing to take account of students, teachers, and their relation with computers; instead of concentrating us on the development of a robust grammar checker for second-language writers.</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
3.1 Pilot Study
</SectionTitle>
      <Paragraph position="0"> The aim pursued in this study was to further explore the relation between second language writers' needs and the possibilities of a language tool such as Granska. The pilot study investigated the use of the language tool by three-second language writers working as researchers at a Swedish University (Knutsson et al., 2002). This study focused more on the texts (users' products) than on the process of writing second language with the help of a language tool. The study showed that the writers followed the advice provided by the language tool. In particular, they found detection and correction feedback more helpful than the diagnosis feature. The analysis of the writers' texts comprising 2700 words contained totally 223 errors.</Paragraph>
      <Paragraph position="1"> Focused on the errors found, we discovered that the language tool detected 36,8% of all errors and proposed corrections for about 34,1% of all errors. The writers detected a few of the errors without requiring the language tool's advice and repaired some of the errors where the language tool provided detection and diagnosis. About 15 false alarms occurred and they were mostly due to the program's limited knowledge of idiomatic expressions.</Paragraph>
      <Paragraph position="2"> None of these alarms caused the writers to make changes in their texts. When looking at the writers' comments they seem to convey a sense of getting annoyed when the program continues to detect and provide diagnosis on sentences without for example, finite verbs. The context in which we conducted this study was a naturalistic academic working place and that represented a compromise.</Paragraph>
      <Paragraph position="3"> The users were primarily doing their work and not testing a tool in a controlled context, they were writing under stress, disposing of little time to systematically save their drafts or evaluate the language tool in use. Their principal goal of their activity was not learning about the language tool and their errors but rather to be able to communicate written information as proper as possible. The collection of data was difficult to gather systematically and that lead us to explore the information collected rather than to conduct comparisons or more refined analysis. Based on these preliminary results, we decided to conduct a study in another naturalistic context with a easier access to the data. Study 1 was conducted with a group of students attending an advanced course in &amp;quot;Writing Swedish as a Foreign Language&amp;quot; at the Stockholm University.</Paragraph>
    </Section>
    <Section position="3" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
3.2 Study 1
</SectionTitle>
      <Paragraph position="0"> The study focused on the process of second-language writing with a language tool. In particular, we aim at analyzing the use of the same language tool by a larger number of writers who were focused more on the process of language learning than on delivering a text. The contact with the National Program &amp;quot;Swedish as a Foreign Language&amp;quot; at the department of Scandinavian languages, Stockholm University, made possible the study of the use of Granska by a group of 20 students. Of the initial 20 students 7 accepted to participate in the study and 5 have actually completed it.</Paragraph>
      <Paragraph position="1"> The research questions guiding this study were: * What do second-language writers need from a language tool? * Which types of mistakes do second language writers often commit? Which are the ones that the language tool better support? * How do second language writers handle false alarms generated by the program? * How much data is it necessary to analyze when studying the use of language tool in a second language learning environment? * How does the teacher regard the use of the language tool in her classroom?</Paragraph>
    </Section>
    <Section position="4" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
Naturalistic Task
</SectionTitle>
      <Paragraph position="0"> We had access to the writers' texts that were part of the regular course on writing Swedish as a second-language.</Paragraph>
      <Paragraph position="1"> The texts consisted of different genres of texts (argumentative texts, letters, descriptions, essays etc.) and on different subjects. The learners composed all texts at home and discussed them at the university. The teacher reviewed their texts and graded them.</Paragraph>
      <Paragraph position="2"> About the Users All the users were in their third and final semester of the language-learning program &amp;quot;Svenska som fr&amp;quot;ammande spr@ak&amp;quot; - Swedish as a foreign language - This program prepares learners to pass the TISUS (Test I Svenska f&amp;quot;or Universitets- och h&amp;quot;ogskoleStudier) a test that is equivalent to the TOEFL (Test Of English as a Foreign Language). All the participants had in average resided in Sweden for a period of three years. They come from different parts of the world: Spain, Germany, Russia, Poland and Philippines, see table 1. They presented mixed background although all had received university education in their mother tongues. They also presented diverse degrees of familiarity with computers. All the participants succeeded the course.</Paragraph>
    </Section>
    <Section position="5" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
Method and Data Collection
</SectionTitle>
      <Paragraph position="0"> Study 1 was conducted in the context of a second-language course. To study the use of a language tool in a real setting entails, first of all, to introduce the tool into the new context. This task requires a lot of effort and entails different steps in a complex process:  1. Establishing contact with the teacher. Learning about the course, its goals, its participants, and its tasks.</Paragraph>
      <Paragraph position="1"> 2. Introducing ourselves and the research project to the course.</Paragraph>
      <Paragraph position="2"> 3. Presenting and explaining the instructions to use and judge Granska. Choosing together with the teacher the text to be revised by the learners using the tool. 4. Distributing the consents forms and the prequestionnaires to those willing to participate in the study.</Paragraph>
      <Paragraph position="3"> 5. Observing the learners using Granska while they revised their texts at the computer lab at the university. Providing help if needed. Collecting their drafts. 6. Checking if the students had problems when interacting with the tool. Providing help if needed. 7. Collecting their final versions and interviewing the learners and the teacher.</Paragraph>
      <Paragraph position="4">  Establishing and developing a relationship with the teacher was fundamental in order to gain her confidence and thus have the chance to introduce the tool into the classroom. It is the teacher who actually decides on the advantages to let the students use a computer program in their composition tasks. Planning together with the teacher the introduction of the computer program into the classroom entailed choosing the class, the writing task and the time of the year in which the language tool could be presented to the students.</Paragraph>
      <Paragraph position="5"> We collected data through: participant observation, students' texts, judgment procedure, questionnaires and interviews. In this study, we especially encouraged writers to use the language tool outside the class. The prerequisite announced for the users was the following: &amp;quot;use the language tool whenever you want and when you feel it will help you&amp;quot;. The control of the data collection was thus left to the users. According to the instructions the user should save the original text scrutinized with Granska and also the final version, written after the revision aided by Granska.</Paragraph>
      <Paragraph position="6"> One important part of the data collection consisted of text material. This material consisted of the output from Granska, the final version of text, and user judgments on the alarms from Granska.</Paragraph>
      <Paragraph position="7"> Judgment procedure The judgment procedure was developed in the pilot study. The purpose of the judging procedure was to track the users' decision when prompted with alarms from Granska. The users were instructed to use the following  grading scale: * Grade 5. Excellent - I understand exactly what Granska suggests.</Paragraph>
      <Paragraph position="8"> * Grade 4. Good - Granska is a quite good help for me.</Paragraph>
      <Paragraph position="9"> * Grade 3. Acceptable - It is hard for me to make up my mind on what Granska says, but I take a chance that Granska is right.</Paragraph>
      <Paragraph position="10"> * Grade 2. Bad - It is hard for me to make up my mind on what Granska says, I have to look in my grammar book. With the help of the book I can decide if I should follow Granska or not.</Paragraph>
      <Paragraph position="11"> * Grade 1. Incomprehensible - I do not understand what Granska says. I have to ask the teacher or some other competent person for help.</Paragraph>
      <Paragraph position="12">  The users were requested to use the grading scale in order to judge the detections, diagnoses and correction proposals from Granska. Definitions of detections, diagnoses and corrections were given. Every scrutinized text was supposed to be annotated with user judgments on the alarms from the program. The annotations should preferably be made electronically, but it was also possible for the users to print out the output from Granska, and make their annotations on paper.</Paragraph>
    </Section>
    <Section position="6" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
3.3 Data Analysis
</SectionTitle>
      <Paragraph position="0"> About the texts and the errors The texts collected were limited in size, although we could identify some tendencies in the texts. The texts were not full of errors. The clauses and sentences were very often quite well formed, and in most cases, they have a subject and main verb structure; in other words, they are quite close to the norm. One finding was that the group of learners chosen, constituted a adequate target group for the use of language tools such as Granska.</Paragraph>
      <Paragraph position="1"> Granska detected about 35% of all errors. An error typology together with Granska's detections, diagnoses and corrections is presented in table 2.</Paragraph>
      <Paragraph position="2"> Most of the undetected errors were syntactical errors.</Paragraph>
      <Paragraph position="3"> These syntactical errors were mostly word order errors, missing words, or problems with choice of preposition in certain verb frames. Granska's recall on target errors was quite good, so the main focus for our future efforts should contain methods for the detection of currently unrecognized errors, and to support the user when she tries to incorporate Granska's advice into her writing process.</Paragraph>
      <Paragraph position="4"> One serious problem was collecting the final versions of students' texts. The final text version actually tracks the user decisions when prompted with Granska's alarms.</Paragraph>
      <Paragraph position="5"> Even students that did not participate in the study were very glad about the judgment procedure; but they were not willing to give us the final versions of their texts.</Paragraph>
      <Paragraph position="6"> Maybe they thought that the final version was something between them and the teacher.</Paragraph>
      <Paragraph position="7"> Another problem was the authentic value of the texts.</Paragraph>
      <Paragraph position="8"> A large amount of data seemed when analyzed to be false; one writer has copied a lot of text from a book. This made us of aware of the fact that only personal, often argumentative texts could be part of these kinds of studies. The results presented in this study consist mostly of argumentative texts.</Paragraph>
      <Paragraph position="9"> Regarding the judgment procedure, we observed that the user judgments gave important cues to the understanding of the user's needs during the revision process and seemed not to disturb the user too much. Instead, the users seemed to appreciate this task (see interviews). See table 3 for some results on the user judgments.</Paragraph>
      <Paragraph position="10"> The analysis of the interviews The analysis of the participants' answers allowed us to better understand how they experienced the use of the language tool in their learning contexts.</Paragraph>
      <Paragraph position="11"> The participants observed presented mixed levels of familiarity with computers programs. Some were keen about using computers and showed experience in using Microsoft Word, e-mail programs while others presented problems in attaching texts to mails or saving files in other formats. The heterogeneity of their computer literacy made it clear the necessity to introduce a course on basic knowledge about using computers.</Paragraph>
      <Paragraph position="12"> Participants found it difficult to interact with the program due to: - false alarms, - the number and type of language explanations to select when correcting a language error, - misunderstandings about error indication provided by the program, - lack of experience in working with more than one window open on the computer, - changing formats and sending documents by e-mail.</Paragraph>
      <Paragraph position="13"> Participants enjoyed using the program due to: - the different colors indicating different steps in the revision of the language, - the possibility to get access to different explanations, - the linguistic terms that the program employs, although some of them resulted difficult to fully understand.</Paragraph>
      <Paragraph position="14"> When asking participants about how we could improve the presentation of the program to other groups of students, they said that we should consider to:  - include more computer training at the beginning of the study - give a more active role to the teacher before and during the study, - remind participants on their duty to send their material, - challenge participants as much as possible.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>