<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-3253">
  <Title>Sentiment analysis using support vector machines with diverse information sources</Title>
  <Section position="6" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
5 Results
</SectionTitle>
    <Paragraph position="0"> The accuracy value represents the percentage of test texts which were classified correctly by the model.</Paragraph>
    <Paragraph position="1"> Results on the first dataset, without topic information, are shown in figure 2. The results for 3-fold cross validation show how the present feature sets compare with the best performing SVM reported in Pang et al.</Paragraph>
    <Paragraph position="2"> In general, the addition of Osgood values does not seem to yield improvement in any of the mod- null software/TinySVM is not surprising given their superior performance alone. In the case of the SVM with only a single Turney value, accuracy is already at 68.3% (Turney (2002) reports that simply averaging these values on the same data yields 65.8% accuracy). The Osgood values are considerably less reliable, yielding only 56.2% accuracy on their own. Lemmas out-perform unigrams in all experiments, and in fact the simple lemma models outperform even those augmented with the Turney and Osgood features in the experiments on the epinions data. The contribution of these new feature types is most pronounced when they are used to train a separate SVM and the two SVMs are combined in a hybrid SVM. The best results are obtained using such hybrid SVMs, which yield scores of 84.6% accuracy on the 3-fold experiments and 86.0% accuracy on the 10-fold experiments. null In the second set of experiments, again, inclusion of Osgood features shows no evidence of yielding any improvement in modeling when other features are present, indeed, as in the previous experiments there are some cases in which these features may be harming performance. The PMI values, on the other hand, appear to yield consistent improvement. Furthermore on both the 20 and 100-fold test suites the inclusion of all PMI values with lemmas outperforms the use of only the Turney values, suggesting that the incorporation of the available topic relations is helpful. Although there is not enough data here to be certain of trends, it is intuitive that the broader PMI values, similarly to the unigrams, would particularly benefit from increased training data, due to their specificity, and therefore their relative sparse-Model 5 folds 10 folds 20 folds 100 folds  review data, hand-annotated for topic. Note that the results for the Turney Values-only model were obtained using a polynomial kernel. All others were obtained with a linear kernel. ness. Once again, the information appears to be most fruitfully combined by building SVMs representing semantic values and lemmas separately and combining them in a single hybrid SVM. The average score over the four n-fold cross validation experiments for the hybrid SVM is 86.5%, whereas the average score for the second-best performing model, incoporating all semantic value features and lemmas, is 85%. The simple lemmas model obtains an average score of 84% and the simple unigrams model obtains 79.75%.</Paragraph>
  </Section>
  <Section position="7" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
6 Discussion
</SectionTitle>
    <Paragraph position="0"> The main development presented here is the incorporation of several new information sources as features into SVMs which previously relied entirely on the effective but limited &amp;quot;bag of words&amp;quot; approach. The ability of SVMs to handle real-valued features makes this possible, and the information sources introduced in the work Turney and Kamps and Marx provide sensible places to start. The intuition that topic relations and proximity should also yield gains also appears to be borne out in the present experiments. The various sources of information appear to be best combined by integrating several distinct SVMs.</Paragraph>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
6.1 Other issues
</SectionTitle>
      <Paragraph position="0"> At the level of the phrasal SO assignment, it would seem that some improvement could be gained by adding domain context to the AltaVista Search.</Paragraph>
      <Paragraph position="1"> Many--perhaps most--terms' favorability content depends to some extent on their context. As Turney notes, &amp;quot;unpredictable,&amp;quot; is generally positive when describing a movie plot, and negative when describing an automobile or a politician. Likewise, such terms as &amp;quot;devastating&amp;quot; might be generally negative, but in the context of music or art may imply an emotional engagement which is usually seen as positive. Likewise, although &amp;quot;excellent&amp;quot; and &amp;quot;poor&amp;quot; as the poles in assessing this value seems somewhat arbitrary, cursory experiments in adjusting the search have thus far supported Turney's conclusion that the former are the appropriate terms to use for this task.</Paragraph>
      <Paragraph position="2"> One problem with limiting the domain by adding topic-related word constraints to the query is that the resultant hit count is greatly diminished, canceling out any potential gain. It is to be hoped that in the future, as search engines continue to improve and the Internet continues to grow, more possibilities will open up in this regard.</Paragraph>
      <Paragraph position="3"> It also seems likely that the topic-relations aspect of the present research only scratches the surface of what should be possible. There is still considerable room for improvement in performance. The present models may also be further expanded with features representing other information sources, which may include other types of semantic annotation (Wiebe, 2002), or features based on more sophisticated grammatical or dependency relations or on zone information. In any case, it is hoped that the present work may help to indicate how various information sources pertinent to the task may be brought together.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>