<?xml version="1.0" standalone="yes"?>
<Paper uid="N06-2027">
  <Title>Using Semantic Authoring for Blissymbols Communication Boards</Title>
  <Section position="5" start_page="106" end_page="107" type="evalu">
    <SectionTitle>
4 Evaluation
</SectionTitle>
    <Paragraph position="0"> We evaluate our system as an AAC application for message generation from communication boards.</Paragraph>
    <Paragraph position="1"> From an NLG evaluation perspective, this corresponds to an intrinsic evaluation, i.e. judging quality criteria of the generated text and its adequacy relative to the input (Bangalore et al., 1998). Since the prototype of our system is not yet adjusted to interact with alternative pointing devices, we could not test it on actual Bliss users, and could not perform a full extrinsic (task-based) evaluation.</Paragraph>
    <Paragraph position="2"> However, as argued in (Higginbotham, 1995), evaluations of AAC systems with nondisabled subjects, when appropriately used, is easier to perform, and in some cases provide superior results.</Paragraph>
    <Paragraph position="3"> Higginbotham's claims rely on the observation that the methods of message production are not unique to AAC users and analogous communication situations exist both for disabled and nondisabled users. Nondisabled subjects can contribute to the understanding of the cognitive processes underlying the acquisition of symbol and device performance competencies. We believe that the evaluation of efficiency for non-AAC users should be served as baseline. null The approach we offer for message generation requires users to plan their sentences abstractly. (Mc-Coy and Hershberger, 1999) points that novel systems may be found to slow communication but to increase literacy skills. We therefore tested both speed of message generation and semantic coverage (the capability to generate a given message correctly).</Paragraph>
    <Paragraph position="4"> The usage of semantic authoring was evaluated on nondisabled subjects through a user study of 10 subjects. This provides a reliable approximation of the learning curve and usability of the system in general (Biller et al., 2005).</Paragraph>
    <Paragraph position="5"> In order to evaluate the keystroke savings of the system we have collected a set of 19 sentences written in Bliss and their full English correspondents. We compared the number of the words in the English sentences with the number of choices needed to generate the sentence with our system. The total number of choice steps is 133, while the total num- null ber of words in the sentences is 122. This simple ratio shows no improvement of keystrokes saving using our system. Savings, therefore, must be calculated in terms of narrowing the choice possibilities in each step of the process.</Paragraph>
    <Paragraph position="6"> However, counting the number of words does not include morphology which in Bliss symbols requires additional choices. We have counted the words in the sentences considering morphology markers of inflections as additional words, all summing to 138, as was suggested in (McCoy and Hershberger, 1999).</Paragraph>
    <Paragraph position="7"> Assuming a display with 50 symbols (and additional keys for functions) - a vocabulary of requires 50 different screens. Assuming symbols are organized by frequencies (first screens present the most frequently used words) or by semantic domain.</Paragraph>
    <Paragraph position="8"> The overall number of selections is reduced using our communication board since the selectional restrictions narrow the number of possible choices that can be made at each step. The extent to which selection time can be reduced at each step depends on the application domain and the ontology structure. We cannot evaluate it in general, but expect that a well-structured ontology could support efficient selection mechanisms, by grouping semantically related symbols in dedicated displays.</Paragraph>
    <Paragraph position="9"> In addition, the semantic authoring approach can generate fluent output in other languages (English and Hebrew, beyond the Bliss sequence - without requiring noisy translation). We also hypothesize that ontologically motivated grouping of symbols could speed up each selection step - but this claim must be assessed empirically in a task-based extrinsic evaluation, which remains to be done in the future.</Paragraph>
    <Paragraph position="10"> We are now building the environment for AAC users with cooperation with ISAAC-ISRAEL 2, in order to make the system fully accessible and to be tested by AAC-users. However, this work is still in progress. Once this will be achieved, full evaluation of the system will be plausible.</Paragraph>
  </Section>
class="xml-element"></Paper>