<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-0839">
  <Title>Complementarity of Lexical and Simple Syntactic Features: The SyntaLex Approach to SENSEVAL-3</Title>
  <Section position="5" start_page="0" end_page="0" type="concl">
    <SectionTitle>
5 Discussion
</SectionTitle>
    <Paragraph position="0"> Observe that even though SyntaLex-2 uses a larger context than SyntaLex-1 it does not do much better than the latter, in fact, its accuracies are slightly lower. We believe this is due to the low training data per task ratio, which usually means that the weak indicators (Pa3a5a2 and Pa2 ) are likely to be overwhelmed by idiosyncrasies of the data. (Mohammad and Pedersen, 2004) show results to the same conclusions for SENSEVAL-1 and SENSEVAL-2 data that have similar low training data per task, while, the line, hard, serve and interest data which have much larger training data per task are shown to benefit from a larger context.</Paragraph>
    <Paragraph position="1"> Duluth-ELSS (a sister system of SyntaLex) achieves an accuracy of 61.7%. It creates an ensemble of three bagged decision trees, where one tree is based on unigrams, another on bigrams, and a third on co-occurrences with the target word. Observe that its accuracy is comparable to SyntaLex-2 (62.4%) which use only Part of Speech features.</Paragraph>
    <Paragraph position="2"> However, these results alone do not tell us if both kinds of features disambiguate the same set of instances correctly, that is, they are mutually redundant, or they classify differing sets of instances correctly, that is, they are mutually complementary.</Paragraph>
    <Paragraph position="3"> Significant complementarity implies that a marked increase in accuracies may be achieved by suitably combining the bigram and Part of Speech features.</Paragraph>
    <Paragraph position="4"> We have shown earlier (Mohammad and Pedersen, 2004) that there is indeed large complementarity between lexical and syntactic features by experiments on line, hard, serve, interest, SENSEVAL-1 and SENSEVAL-2 data. We use the measures Optimal Ensemble and Baseline Ensemble, introduced there, to quantify the complementarity and redundancy between bigrams and Part of Speech features in the SENSEVAL-3 data.</Paragraph>
    <Paragraph position="5"> The Baseline Ensemble of bigram and PoS features is the accuracy of a hypothetical ensemble that correctly disambiguates an instance only when the individual classifiers based on both kinds of features correctly identify the intended sense. The Optimal Ensemble of bigrams and PoS features is the accuracy of a hypothetical ensemble that accurately disambiguates an instance when any of the two individual classifiers correctly disambiguates the intended sense. We find the Baseline Ensemble of bigrams and PoS features on SENSEVAL-3 data to be 52.9% and the Optimal Ensemble to be 72.1%. Thus, given 100 instances, almost 53 of them would be correctly tagged by both kinds of classifiers and up to 72 may be correctly disambiguated using a powerful ensemble technique.</Paragraph>
    <Paragraph position="6">  In order to capitalize on the significant complementarity of bigrams and Part of Speech features, SyntaLex-3 uses a simple ensemble technique, while SyntaLex-4 learns a unified decision tree based on both bigrams and Part of Speech features. Observe that both SyntaLex-3 and 4 achieve accuracies higher than SyntaLex-1 and  2. Further, SyntaLex-3 performs slightly better than SyntaLex-4. We believe that SyntaLex-4 may be affected by data fragmentation caused by  learning decision trees from a large number of features and limited training data. We also note that the Optimal Ensemble is markedly higher than the accuracies of SyntaLex-3and 4, suggesting that the use of a more powerful combining methodology is justified.</Paragraph>
  </Section>
class="xml-element"></Paper>