<?xml version="1.0" standalone="yes"?>
<Paper uid="W05-0210">
  <Title>amp; Spoken Language Communication Research Laboratories</Title>
  <Section position="4" start_page="65" end_page="66" type="metho">
    <SectionTitle>
5 Discussion
</SectionTitle>
    <Paragraph position="0"> This section explains the on-demand generation of FBQs according to individual preference, an immediate extension and a limitation of our proposed method, and finally touches on free-format Q&amp;A.</Paragraph>
    <Paragraph position="1"> Effects of Automatic FBQ Construction The method provides teachers and testers with a tool that reduces time and expenditure. Furthermore, the method can deal with any text. For example, up-to-date and interesting materials such as news articles of the day can be a source of seed sentences (Figure 6 is a sample generated from an article (http://www.japantimes.co.jp/) on an earthquake that occurred in Japan), which enables realization of a personalized learning environment.</Paragraph>
    <Paragraph position="2">  question from a Web news article in The Japan Times on &amp;quot;an earthquake&amp;quot; N.B. The correct answer is c) originated.</Paragraph>
    <Paragraph position="3"> Question 2 (FBQ) The second quake 10 km below the seabed some 130 km east of Cape Shiono.</Paragraph>
    <Paragraph position="4"> a) put b) came c) originated d) opened We have generated questions from over 100 documents on various genres such as novels, speeches, academic papers and so on found in the enormous collection of e-Books provided by Project Gutenberg (http://www.gutenberg.org/).</Paragraph>
    <Paragraph position="5"> Proficiency th estimated with the reduced test and its relation to TOEIC Scores Figure 5 shows the relationship between reduction of the test size according to the method explained in Section 3.2 and the estimated proficiency based on the reduced test. The x-axis represents the size of the reduced test in number of items, while the y-axis represents the correlation coefficient (R) between estimated proficiency and real TOEIC score. A Variation of Fill-in-the-Blank Questions for Grammar Checking In Section 2.2, we mentioned a constraint that a good distracter should maintain the grammatical characteristics of the correct choice originating in  the seed sentence. The question checks not the grammaticality but the semantic/pragmatic correctness. null We can generate another type of FBQ by slightly modifying step [b] of the procedure in Section 2.2 to retain the stem of the original word w and vary the surface form of the word w. This modified procedure generates a question that checks the grammatical ability of the test takers.  Limitation of the Addressed FBQs The questions dealt with in this paper concern testing reading ability, but these questions are not suitable for testing listening ability because they are presented visually and cannot be pronounced. To test listening ability, like in TOIEC, other types of questions should be used, and automated generation of them is yet to be developed.</Paragraph>
    <Paragraph position="6"> Free-Format Q&amp;A Besides measuring one's ability to receive information in a foreign language, which has been addressed so far in this paper, it is important to measure a person's ability to transmit information in a foreign language. For that purpose, tests for translating, writing, or speaking in a free format have been actively studied by many researchers (Shermis, 2003; Yasuda, 2004).</Paragraph>
    <Section position="1" start_page="66" end_page="66" type="sub_section">
      <SectionTitle>
Related Work
</SectionTitle>
      <Paragraph position="0"> * Here, we explain other studies on the generation of multiple-choice questions for language learning. There are a few previous studies on computer-based generation such as Mitkov (2003) and Wilson (1997).</Paragraph>
    </Section>
  </Section>
  <Section position="5" start_page="66" end_page="67" type="metho">
    <SectionTitle>
7 Conclusion
Cloze Test
</SectionTitle>
    <Paragraph position="0"> A computer can generate questions by deleting words or parts of words randomly or at every N-th word from text. Test-takers are requested to restore the word that has been deleted. This is called a &amp;quot;cloze test.&amp;quot; The effectiveness of a &amp;quot;cloze test&amp;quot; or its derivatives is a matter of controversy among researchers of language testing such as Brown (1993) and Alderson (1996).</Paragraph>
    <Paragraph position="1"> N.B. The correct answer is c) care.</Paragraph>
    <Paragraph position="2"> Question 3 (FBQ) Because the equipment is very delicate, it must be handled with ______.</Paragraph>
    <Paragraph position="3"> a) caring b) careful c) care d) carefully Tests on Facts Mitkov (2003) proposed a computer-aided procedure for generating multiple-choice questions from textbooks. The differences from our proposal are that (1) Mitkov's method generates questions not about language usage but about facts explicitly stated in a text + ; (2) Mitkov uses techniques such as term extraction, parsing, transformation of trees, which are different from our proposal; and (3) Mitkov does not use IRT while we use it.</Paragraph>
    <Paragraph position="4"> This paper proposed the automatic construction of Fill-in-the-Blank Questions (FBQs). The proposed method generates FBQs using a corpus, a thesaurus, and the Web. The generated questions and Item Response Theory (IRT) then estimate second-language proficiency.</Paragraph>
    <Paragraph position="5"> Experiments have shown that the proposed method is effective in that the estimated proficiency highly correlates with non-native speakers' real proficiency as represented by TOEIC scores; native-speakers can achieve higher scores than non-native speakers. It is possible to reduce the size of the test by removing non-discriminative questions with item information in IRT.</Paragraph>
    <Paragraph position="6"> + Based on a fact stated in a textbook like, &amp;quot;A prepositional phrase at the beginning of a sentence constitutes an introductory modifier,&amp;quot; Mitkov generates a question such as, &amp;quot;What does a prepositional phrase at the beginning of a sentence constitute? i. a modifier that accompanies a noun; ii. an associated modifier; iii. an introductory modifier; iv. a misplaced modifier.&amp;quot; * There are many works on item generation theory (ITG) such as Irvine and Kyllonen (2002), although we do not go any further into the area. We focus only on multiple-choice questions for language learning in this paper.</Paragraph>
    <Paragraph position="7">  The method provides teachers, testers, and test takers with novel merits that enable low-cost testing of second-language proficiency and provides learners with up-to-date and interesting materials suitable for individuals.</Paragraph>
    <Paragraph position="8"> Further research should be done on (1) large-scale evaluation of the proposal, (2) application to different languages such as Chinese and Korean, and (3) generation of different types of questions.</Paragraph>
  </Section>
class="xml-element"></Paper>