<?xml version="1.0" standalone="yes"?>
<Paper uid="P06-1086">
  <Title>MAGEAD: A Morphological Analyzer and Generator for the Arabic Dialects</Title>
  <Section position="13" start_page="686" end_page="686" type="concl">
    <SectionTitle>
8.4 Qualitative Analysis: MSA
</SectionTitle>
    <Paragraph position="0"> The gold standard we are using has been generated automatically using the Buckwalter analyzer.</Paragraph>
    <Paragraph position="1"> Only the contextually correct analysis has been hand-checked. As a result, our quantitative analysis in Section 8.3 leaves open the question of how good the gold standard is in the first place. We analyzed all of the 2,536 false positives (types) produced by MAGEAD on our development set (analyses it suggested, but which the Test corpus did not have). In 75% of the errors, the Buckwalter analyzer does not provide a passive voice analysis which differs from the active voice one only in diacritics which are not written. 7% are cases where Buckwalter does not make distinctions that MAGEAD makes (e.g. mood variations that are not phonologically realized); in 4.4% of the errors a correct analysis was created but it was not produced by Buckwalter for various reasons. If we count these cases as true positives rather than as false positives (as in the case in Figure 1) and take type frequency into account, we obtain a token precision rate of 94.9% on the development set.</Paragraph>
    <Paragraph position="2"> The remaining cases are MAGEAD errors. 3.3% are missing rules to handle special cases such as jussive mood interaction with weak radicals; 5.4% are incorrect combinations of morphemes such as passive voice and object pronouns; 2.6% of the errors are cases of pragmatic overgeneration such as second person masculine subjects with a second person feminine plural object. 1.5% of the errors are errors of the mbc-root list and 1.2% are other errors. A large number of these errors are fixable errors.</Paragraph>
    <Paragraph position="3"> There were 162 false negatives (gold standard analyses MAGEAD did not get). 65.4% of these errors were a result of the use of the mbc list restriction. The rest of the errors are all a result of unhandled phenomena in MAGEAD: quadrilateral roots (13.6%), imperatives (8%), and specific missing rules/ rule failures (13%) (e.g., for handling some weak radicals/hamza cases, pattern IX gemination-like behavior, etc.).</Paragraph>
    <Paragraph position="4"> We conclude that we can claim that our precision numbers are actually much higher, and that we can further improve them by adding more rules and knowledge to MAGEAD.</Paragraph>
    <Section position="1" start_page="686" end_page="686" type="sub_section">
      <SectionTitle>
8.5 Quantitative and Qualitative Analysis:
Levantine
</SectionTitle>
      <Paragraph position="0"> For the Levantine, we do not have a list of all possible analyses for each word in the gold standard: only the contextually appropriate analysis is hand-checked. We therefore only report context recall in Figure 2. As a baseline, we report the MSA MAGEAD with the all restriction applied to the same Levantine test corpus. As we can see, the MSA system performs poorly on Levantine input. The Levantine system we use is the one described in Section 7. We use the resulting analyzer with the all option as we have no information on roots in Levantine. MAGEAD with Levantine knowledge does well, missing only one in 20 contextually correct analyses. We take this to mean that the architecture of MAGEAD allows us to port MAGEAD fairly rapidly to a new dialect and to perform adequately well on the most important analysis for each token, the contextually relevant one.</Paragraph>
      <Paragraph position="1">  For the Levantine MAGEAD, there were 25 errors, cases of contextually selected analyses that MAGEAD did not get (false negatives). Most of these are related to phenomena that MAGEAD doesn't currently handle: imperatives (48%) (which are much more common in speech corpora) and quadrilateral roots (8%). There were four cases (16%) of an unhandled variant spelling of an object pronoun and 7 cases (28%) of hamza/weak radical rule errors.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>