<?xml version="1.0" standalone="yes"?>
<Paper uid="J04-3002">
  <Title>at Asheville</Title>
  <Section position="5" start_page="301" end_page="303" type="metho">
    <SectionTitle>
5. Relation to Other Work
</SectionTitle>
    <Paragraph position="0"> There has been much work in other fields, including linguistics, literary theory, psychology, philosophy, and content analysis, involving subjective language. As mentioned in Section 2, the conceptualization underlying our manual annotations is based on work in literary theory and linguistics, most directly DoleVzel (1973), Uspensky (1973), Kuroda (1973, 1976), Chatman (1978), Cohn (1978), Fodor (1979), and Banfield (1982). We also mentioned existing knowledge resources such as affective lexicons (General-Inquirer 2000; Heise 2000) and annotations in more general-purpose lexicons (e.g., the attitude adverb features in Comlex [Macleod, Grishman, and Meyers 1998]).</Paragraph>
    <Paragraph position="1"> Such knowledge may be used in future work to complement the work presented in this article, for example, to seed the distributional-similarity process described in Section 3.4.</Paragraph>
    <Paragraph position="2"> There is also work in fields such as content analysis and psychology on statistically characterizing texts in terms of word lists manually developed for distinctions related to subjectivity. For example, Hart (1984) performs counts on a manually developed list of words and rhetorical devices (e.g., &amp;quot;sacred&amp;quot; terms such as freedom) in political speeches to explore potential reasons for public reactions. Anderson and McMaster (1998) use fixed sets of high-frequency words to assign connotative scores to documents and sections of documents along dimensions such as how pleasant, acrimonious, pious, or confident, the text is.</Paragraph>
    <Paragraph position="3"> What distinguishes our work from work on subjectivity in other fields is that we focus on (1) automatically learning knowledge from corpora, (2) automatically  Computational Linguistics Volume 30, Number 3 performing contextual disambiguation, and (3) using knowledge of subjectivity in NLP applications. This article expands and integrates the work reported in Wiebe and Wilson (2002), Wiebe, Wilson, and Bell (2001), Wiebe et al. (2001) and Wiebe (2000). Previous work in NLP on the same or related tasks includes sentence-level and document-level subjectivity classifications. At the sentence level, Wiebe, Bruce, and O'Hara (1999) developed a machine learning system to classify sentences as subjective or objective. The accuracy of the system was more than 20 percentage points higher than a baseline accuracy. Five part-of-speech features, two lexical features, and a paragraph feature were used. These results suggested to us that there are clues to subjectivity that might be learned automatically from text and motivated the work reported in the current article. The system was tested in 10-fold cross validation experiments using corpus WSJ-SE, a small corpus of only 1,001 sentences. As discussed in Section 1, a main goal of our current work is to exploit existing document-level annotations, because they enable us to use much larger data sets, they were created outside our research group, and they allow us to assess consistency of performance by cross-validating between our manual annotations and the existing document-level annotations. Because the document-level data are not annotated at the sentence level, sentence-level classification is not highlighted in this article. The new sentence annotation study to evaluate sentences with high-density features (Section 4.5) uses different data from WSJ-SE, because some of the features (n-grams and density parameters) were identified using WSJ-SE as training data.</Paragraph>
    <Paragraph position="4"> Other previous work in NLP has addressed related document-level classifications.</Paragraph>
    <Paragraph position="5"> Spertus (1997) developed a system for recognizing inflammatory messages. As mentioned earlier in the article, inflammatory language is a type of subjective language, so the task she addresses is closely related to ours. She uses machine learning to select among manually developed features. In contrast, the focus in our work is on automatically identifying features from the data.</Paragraph>
    <Paragraph position="6"> A number of projects investigating genre detection include editorials as one of the targeted genres. For example, in Karlgren and Cutting (1994), editorials are one of fifteen categories, and in Kessler, Nunberg, and Sch &amp;quot;utze (1997), editorials are one of six. Given the goal of these works to perform genre detection in general, they use low-level features that are not specific to editorials. Neither shows significant improvements for editorial recognition. Argamon, Koppel, and Avneri (1998) address a slightly different task, though it does involve editorials. Their goal is to distinguish not only, for example, news from editorials, but also these categories in different publications. Their best results are distinguishing among the news categories of different publications; their lowest results involve editorials. Because we focus specifically on distinguishing opinion pieces from nonopinion pieces, our results are better than theirs for those categories. In addition, in contrast to the above studies, the focus of our work is on learning features of subjectivity. We perform opinion piece recognition in order to assess the usefulness of the various features when used together.</Paragraph>
    <Paragraph position="7"> Other previous NLP research has used features similar to ours for other NLP tasks.</Paragraph>
    <Paragraph position="8"> Low-frequency words have been used as features in information extraction (Weeber, Vos, and Baayen 2000) and text categorization (Copeck et al. 2000). A number of researchers have worked on mining collocations from text to extend lexicographic resources for machine translation and word sense disambiguation (e.g., Smajda 1993; Lin 1999; Biber 1993).</Paragraph>
    <Paragraph position="9"> In Samuel, Carberry, and Vijay-Shanker's (1998) work on identifying collocations for dialog-act recognition, a filter similar to ours was used to eliminate redundant n-gram features: n-grams were eliminated if they contained substrings with the same entropy score as or a better entropy score than the n-gram.</Paragraph>
    <Paragraph position="10">  Wiebe, Wilson, Bruce, Bell, and Martin Learning Subjective Language While it is common in studies of collocations to omit low-frequency words and expressions from analysis, because they give rise to invalid or unrealistic statistical measures (Church and Hanks, 1990), we are able to identify higher-precision collocations by including placeholders for unique words (i.e., the ugen-n-grams). We are not aware of other work that uses such collocations as we do.</Paragraph>
    <Paragraph position="11"> Features identified using distributional similarity have previously been used for syntactic and semantic disambiguation (Hindle 1990; Dagan, Pereira, and Lee 1994) and to develop lexical resources from corpora (Lin 1998; Riloff and Jones 1999).</Paragraph>
    <Paragraph position="12"> We are not aware of other work identifying and using density parameters as described in this article.</Paragraph>
    <Paragraph position="13"> Since our experiments, other related work in NLP has been performed. Some of this work addresses related but different classification tasks. Three studies classify reviews as positive or negative (Turney 2002; Pang, Lee, and Vaithyanathan 2002; Dave, Lawrence, Pennock 2003). The input is assumed to be a review, so this task does not include finding subjective documents in the first place. The first study listed above (Turney 2002) uses a variation of the semantic similarity procedure presented in Wiebe (2000) (Section 3.4). The third (Dave, Lawrence, and Pennock 2003) uses n-gram features identified with a variation of the procedure presented in Wiebe, Wilson, and Bell (2001) (Section 3.3). Tong (2001) addresses finding sentiment timelines, that is, tracking sentiments over time in multiple documents. For clues of subjectivity, he uses manually developed lexical rules, rather than automatically learning them from corpora. Similarly, Gordon et al. (2003) use manually developed grammars to detect some types of subjective language. Agrawal et al. (2003) partition newsgroup authors into camps based on quotation links. They do not attempt to recognize subjective language.</Paragraph>
    <Paragraph position="14"> The most closely related new work is Riloff, Wiebe, and Wilson (2003), Riloff and Wiebe (2003) and Yu and Hatzivassiloglou (2003). The first two focus on finding additional types of subjective clues (nouns and extraction patterns identified using extraction pattern bootstrapping). Yu and Hatzivassiloglou (2003) perform opinion text classification. They also use existing WSJ document classes for training and testing, but they do not include the entire corpus in their experiments, as we do. Their opinion piece class consists only of editorials and letters to the editor, and their nonopinion class consists only of business and news. They report an average F-measure of 96.5%.</Paragraph>
    <Paragraph position="15"> Our result of 94% accuracy on document level classification is almost comparable.</Paragraph>
    <Paragraph position="16"> They also perform sentence-level classification.</Paragraph>
    <Paragraph position="17"> We anticipate that knowledge of subjective language may be usefully exploited in a number of NLP application areas and hope that the work presented in this article will encourage others to experiment with subjective language in their applications. More generally, there are many types of artificial intelligence systems for which state-of-affairs types such as beliefs and desires are central, including systems that perform plan recognition for understanding narratives (Dyer 1982; Lehnert et al. 1983), for argument understanding (Alvarado, Dyer, and Flowers 1986), for understanding stories from different perspectives (Carbonell 1979), and for generating language under different pragmatic constraints (Hovy 1987). Knowledge of linguistic subjectivity could enhance the abilities of such systems to recognize and generate expressions referring to such states of affairs in natural text.</Paragraph>
  </Section>
class="xml-element"></Paper>