<?xml version="1.0" standalone="yes"?>
<Paper uid="H05-1043">
  <Title>Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 339-346, Vancouver, October 2005. c(c)2005 Association for Computational Linguistics Extracting Product Features and Opinions from Reviews</Title>
  <Section position="4" start_page="339" end_page="344" type="metho">
    <SectionTitle>
3 OPINE Overview
</SectionTitle>
    <Paragraph position="0"> This section gives an overview of OPINE (see Figure 1) and describes its components and their experimental evaluation. null Goal Given product class C with instances I and reviews R, OPINE's goal is to find a set of (feature, opinions) tuples{(f,oi,...oj)}s.t. f [?]F and oi,...oj [?]O, where: a) F is the set of product class features in R.</Paragraph>
    <Paragraph position="1"> b) O is the set of opinion phrases in R.</Paragraph>
    <Paragraph position="2"> c) f is a feature of a particular product instance. d) o is an opinion about f in a particular sentence. d) the opinions associated with each feature f are ranked based on their strength.</Paragraph>
    <Paragraph position="3"> Solution The steps of our solution are outlined in Figure 1 above. OPINE parses the reviews using MINIPAR (Lin, 1998) and applies a simple pronoun-resolution module to parsed review data. OPINE then uses the data to find explicit product features (E). OPINE's Feature Assessor and its use of Web PMI statistics are vital for the extraction of high-quality features (see 3.2). OPINE then identifies opinion phrases associated with features in E and finds their polarity. OPINE's novel use of relaxation-labeling techniques for determining the semantic orientation of potential opinion words in the context of given features and sentences leads to high precision and recall on the tasks of opinion phrase extraction and opinion phrase polarity extraction (see 3.3).</Paragraph>
    <Paragraph position="4"> In this paper, we only focus on the extraction of explicit features, identifying corresponding customer opinions about these features and determining their polarity. We omit the descriptions of the opinion clustering, implicit feature generation and opinion ranking algorithms. 3.0.1 The KnowItAll System.</Paragraph>
    <Paragraph position="5"> OPINE is built on top of KnowItAll, a Web-based, domain-independent information extraction system (Etzioni et al., 2005). Given a set of relations of interest, KnowItAll instantiates relation-specific generic extraction patterns into extraction rules which find candidate facts. KnowItAll's Assessor then assigns a probability to each candidate. The Assessor uses a form of Point-wise Mutual Information (PMI) between phrases that is estimated from Web search engine hit counts (Turney, 2001). It computes the PMI between each fact and automatically generated discriminator phrases (e.g., &amp;quot;is a scanner&amp;quot; for the isA() relationship in the context of the Scanner class). Given fact f and discriminator d, the computed</Paragraph>
    <Paragraph position="7"> The PMI scores are converted to binary features for a Naive Bayes Classifier, which outputs a probability associated with each fact (Etzioni et al., 2005).</Paragraph>
    <Section position="1" start_page="339" end_page="340" type="sub_section">
      <SectionTitle>
3.1 Finding Explicit Features
</SectionTitle>
      <Paragraph position="0"> OPINE extracts explicit features for the given product class from parsed review data. First, the system recursively identifies both the parts and the properties of the given product class and their parts and properties, in turn, continuing until no candidates are found. Then, the system finds related concepts as described in (Popescu et al., 2004) and extracts their parts and properties. Table 1 shows that each feature type contributes to the set of final features (averaged over 7 product classes).</Paragraph>
      <Paragraph position="1">  In order to find parts and properties, OPINE first extracts the noun phrases from reviews and retains those with frequency greater than an experimentally set threshold. OPINE's Feature Assessor, which is an instantiation of KnowItAll's Assessor, evaluates each noun phrase by computing the PMI scores between the phrase and meronymy discriminators associated with the product class (e.g., &amp;quot;of scanner&amp;quot;, &amp;quot;scanner has&amp;quot;, &amp;quot;scanner comes with&amp;quot;, etc. for the Scanner class). OPINE distinguishes parts from properties using WordNet's IS-A hierarchy (which enumerates different kinds of properties) and morphological cues (e.g., &amp;quot;-iness&amp;quot;, &amp;quot;-ity&amp;quot; suffixes).</Paragraph>
    </Section>
    <Section position="2" start_page="340" end_page="340" type="sub_section">
      <SectionTitle>
3.2 Experiments: Explicit Feature Extraction
</SectionTitle>
      <Paragraph position="0"> In our experiments we use sets of reviews for 7 product classes (1621 total reviews) which include the publicly available data sets for 5 product classes from (Hu and Liu, 2004). Hu's system is the review mining system most relevant to our work. It uses association rule mining to extract frequent review noun phrases as features. Frequent features are used to find potential opinion words (only adjectives) and the system uses Word-Net synonyms/antonyms in conjunction with a set of seed words in order to find actual opinion words. Finally, opinion words are used to extract associated infrequent features. The system only extracts explicit features.</Paragraph>
      <Paragraph position="1"> On the 5 datasets in (Hu and Liu, 2004), OPINE's precision is 22% higher than Hu's at the cost of a 3% recall drop. There are two important differences between OPINE and Hu's system: a) OPINE's Feature Assessor uses PMI assessment to evaluate each candidate feature and b) OPINE incorporates Web PMI statistics in addition to review data in its assessment. In the following, we quantify the performance gains from a) and b).</Paragraph>
      <Paragraph position="2"> a) In order to quantify the benefits of OPINE's Feature Assessor, we use it to evaluate the features extracted by Hu's algorithm on review data (Hu+A/R). The Feature Assessor improves Hu's precision by 6%.</Paragraph>
      <Paragraph position="3"> b) In order to evaluate the impact of using Web PMI statistics, we assess OPINE's features first on reviews (OP/R) and then on reviews in conjunction with the Web (the corresponding methods are Hu+A/R+W and OPINE). Web PMI statistics increase precision by an average of 14.5%.</Paragraph>
      <Paragraph position="4"> Overall, 1/3 of OPINE's precision increase over Hu's system comes from using PMI assessment on reviews and the other 2/3 from the use of the Web PMI statistics.</Paragraph>
      <Paragraph position="5"> In order to show that OPINE's performance is robust across multiple product classes, we used two sets of reviews downloaded from tripadvisor.com for Hotels and amazon.com for Scanners. Two annotators labeled a set of unique 450 OPINE extractions as correct or incorrect. The inter-annotator agreement was 86%.</Paragraph>
      <Paragraph position="6"> The extractions on which the annotators agreed were used to compute OPINE's precision, which was 89%. Fur- null Extraction Task. OPINE's recall is 3% lower than the recall of Hu's original system (precision level = 0.8). All results are reported with respect to Hu's.</Paragraph>
      <Paragraph position="7"> thermore, the annotators extracted explicit features from 800 review sentences (400 for each domain). The inter-annotator agreement was 82%. OPINE's recall on the set of 179 features on which both annotators agreed was 73%.</Paragraph>
    </Section>
    <Section position="3" start_page="340" end_page="340" type="sub_section">
      <SectionTitle>
3.3 Finding Opinion Phrases and Their Polarity
</SectionTitle>
      <Paragraph position="0"> This subsection describes how OPINE extracts potential opinion phrases, distinguishes between opinions and nonopinions, and finds the polarity of each opinion in the context of its associated feature in a particular review sentence. null</Paragraph>
    </Section>
    <Section position="4" start_page="340" end_page="341" type="sub_section">
      <SectionTitle>
3.3.1 Extracting Potential Opinion Phrases
</SectionTitle>
      <Paragraph position="0"> OPINE uses explicit features to identify potential opinion phrases. Our intuition is that an opinion phrase associated with a product feature will occur in its vicinity.</Paragraph>
      <Paragraph position="1"> This idea is similar to that of (Kim and Hovy, 2004) and (Hu and Liu, 2004), but instead of using a window of size k or the output of a noun phrase chunker, OPINE takes advantage of the syntactic dependencies computed by the MINIPAR parser. Our intuition is embodied by 10 extraction rules, some of which are shown in Table 4. If an explicit feature is found in a sentence, OPINE applies the extraction rules in order to find the heads of potential opinion phrases. Each head word together with its modi- null fiers is returned as a potential opinion phrase1.</Paragraph>
      <Paragraph position="3"> the Extraction of Potential Opinion Phrases. Notation: po=potential opinion, M=modifier, NP=noun phrase, S=subject, P=predicate, O=object. Extracted phrases are enclosed in parentheses. Features are indicated by the typewriter font. The equality conditions on the left-hand side use po's head.</Paragraph>
    </Section>
    <Section position="5" start_page="341" end_page="343" type="sub_section">
      <SectionTitle>
Rule Templates Rules
</SectionTitle>
      <Paragraph position="0"> dep(w,wprime) m(w,wprime) [?]v s.t. dep(w,v),dep(v,wprime) [?]v s.t. m(w,v),o(v,wprime) [?]v s.t. dep(w,v),dep(wprime,v) [?]v s.t. m(w,v),o(wprime,v) Table 5: Dependency Rule Templates For Finding Words w, w' with Related SO Labels . OPINE instantiates these templates in order to obtain extraction rules. Notation: dep=dependent, m=modifier, o=object, v,w,w'=words.</Paragraph>
      <Paragraph position="1"> OPINE examines the potential opinion phrases in order to identify the actual opinions. First, the system finds the semantic orientation for the lexical head of each potential opinion phrase. Every phrase whose head word has a positive or negative semantic orientation is then retained as an opinion phrase. In the following, we describe how OPINE finds the semantic orientation of words.</Paragraph>
      <Paragraph position="2">  OPINE finds the semantic orientation of a word w in the context of an associated feature f and sentence s. We restate this task as follows: Task Given a set of semantic orientation (SO) labels ({positive,negative,neutral}), a set of reviews and a set of tuples (w, f, s), where w is a potential opinion word associated with feature f in sentence s, assign a SO label to each tuple (w, f, s).</Paragraph>
      <Paragraph position="3"> For example, the tuple (sluggish, driver, &amp;quot;I am not happy with this sluggish driver&amp;quot;) would be assigned a negative SO label.</Paragraph>
      <Paragraph position="4"> Note: We use &amp;quot;word&amp;quot; to refer to a potential opinion word w and &amp;quot;feature&amp;quot; to refer to the word or phrase which represents the explicit feature f.</Paragraph>
      <Paragraph position="5"> Solution OPINE uses the 3-step approach below:  1. Given the set of reviews, OPINE finds a SO label for each word w.</Paragraph>
      <Paragraph position="6"> 2. Given the set of reviews and the set of SO labels for  finds a SO label for each (w, f, s) input tuple.</Paragraph>
      <Paragraph position="7"> Each of these subtasks is cast as an unsupervised collective classification problem and solved using the same mechanism. In each case, OPINE is given a set of objects (words, pairs or tuples) and a set of labels (SO labels); OPINE then searches for a global assignment of labels to objects. In each case, OPINE makes use of local constraints on label assignments (e.g., conjunctions and disjunctions constraining the assignment of SO labels to words (Hatzivassiloglou and McKeown, 1997)).</Paragraph>
      <Paragraph position="8"> A key insight in OPINE is that the problem of searching for a global SO label assignment to words, pairs or tuples while trying to satisfy as many local constraints on assignments as possible is analogous to labeling problems in computer vision (e.g., model-based matching). OPINE uses a well-known computer vision technique, relaxation labeling (Hummel and Zucker, 1983), in order to solve the three subtasks described above.</Paragraph>
      <Paragraph position="9">  Relaxation labeling is an unsupervised classification technique which takes as input: a) a set of objects (e.g., words) b) a set of labels (e.g., SO labels) c) initial probabilities for each object's possible labels d) the definition of an object o's neighborhood (a set of other objects which influence the choice of o's label) e) the definition of neighborhood features f) the definition of a support function for an object label The influence of an object o's neighborhood on its label L is quantified using the support function. The support function computes the probability of the label L being assigned to o as a function of o's neighborhood features. Examples of features include the fact that a certain local constraint is satisfied (e.g., the word nice participates in the conjunction and together with some other word whose SO label is estimated to be positive).</Paragraph>
      <Paragraph position="10"> Relaxation labeling is an iterative procedure whose output is an assignment of labels to objects. At each iteration, the algorithm uses an update equation to reestimate the probability of an object label based on its previous probability estimate and the features of its neighborhood. The algorithm stops when the global label assignment stays constant over multiple consecutive iterations.</Paragraph>
      <Paragraph position="11"> We employ relaxation labeling for the following reasons: a) it has been extensively used in computer-vision with good results b) its formalism allows for many types of constraints on label assignments to be used simultaneously. As mentioned before, constraints are integrated into the algorithm as neighborhood features which influence the assignment of a particular label to a particular object.</Paragraph>
      <Paragraph position="12"> OPINE uses the following sources of constraints:  a) conjunctions and disjunctions in the review text b) manually-supplied syntactic dependency rule templates (see Table 5). The templates are automatically instantiated by our system with different dependency relationships (premodifier, postmodifier, subject, etc.) in order to obtain syntactic dependency rules which find words with related SO labels.</Paragraph>
      <Paragraph position="13"> c) automatically derived morphological relationships (e.g., &amp;quot;wonderful&amp;quot; and &amp;quot;wonderfully&amp;quot; are likely to have similar SO labels).</Paragraph>
      <Paragraph position="14"> d) WordNet-supplied synonymy, antonymy, IS-A and morphological relationships between words. For example, clean and neat are synonyms and so they are likely to have similar SO labels.</Paragraph>
      <Paragraph position="15"> Each of the SO label assignment subtasks previously identified is solved using a relaxation labeling step. In the following, we describe in detail how relaxation labeling is used to find SO labels for words in the given review sets.</Paragraph>
      <Paragraph position="16"> 3.3.4 Finding SO Labels for Words For many words, a word sense or set of senses is used throughout the review corpus with a consistently positive, negative or neutral connotation (e.g., &amp;quot;great&amp;quot;, &amp;quot;awful&amp;quot;, etc.). Thus, in many cases, a word w's SO label in the context of a feature f and sentence s will be the same as its SO label in the context of other features and sentences. In the following, we describe how OPINE's relaxation labeling mechanism is used to find a word's dominant SO label in a set of reviews.</Paragraph>
      <Paragraph position="17"> For this task, a word's neighborhood is defined as the set of words connected to it through conjunctions, disjunctions and all other relationships previously introduced as sources of constraints.</Paragraph>
      <Paragraph position="18"> RL uses an update equation to re-estimate the probability of a word label based on its previous probability estimate and the features of its neighborhood (see Neighborhood Features). At iteration m, let q(w,L)(m) denote the support function for label L of w and let</Paragraph>
      <Paragraph position="20"> where Lprime [?] {pos,neg,neutral} and a &gt; 0 is an experimentally set constant keeping the numerator and probabilities positive. RL's output is an assignment of dominant SO labels to words.</Paragraph>
      <Paragraph position="21"> In the following, we describe in detail the initialization step, the derivation of the support function formula and the use of neighborhood features.</Paragraph>
      <Paragraph position="22"> RL Initialization Step OPINE uses a version of Turney's PMI-based approach (Turney, 2003) in order to derive the initial probability estimates (P(l(w) = L)(0)) for a subset S of the words. OPINE computes a SO score so(w) for each w in S as the difference between the PMI of w with positive keywords (e.g., &amp;quot;excellent&amp;quot;) and the PMI of w with negative keywords (e.g., &amp;quot;awful&amp;quot;). When so(w) is small, or w rarely co-occurs with the keywords, w is classified as neutral. If so(w) &gt; 0, then w is positive, otherwise w is negative. OPINE then uses the labeled S set in order to compute prior probabilities</Paragraph>
      <Paragraph position="24"> the ratio between the number of words in S labeled L and |S|. Such probabilities are used as initial probability estimates associated with the labels of the remaining words.</Paragraph>
      <Paragraph position="25"> Support Function The support function computes the probability of each label for word w based on the labels of objects in w's neighborhood N.</Paragraph>
      <Paragraph position="26"> Let Ak = {(wj,Lj)|wj [?] N} , 0 &lt; k [?] 3|N |represent one of the potential assignments of labels to the words in N. Let P(Ak)(m) denote the probability of this particular assignment at iteration m. The support for label L of word w at iteration m is :</Paragraph>
      <Paragraph position="28"> We assume that the labels of w's neighbors are independent of each other and so the formula becomes:</Paragraph>
      <Paragraph position="30"> probability that l(wj) = Lj (which was computed at iteration m using the RL update equation).</Paragraph>
      <Paragraph position="31"> The P(l(w) = L|Ak)(m) term quantifies the influence of a particular label assignment to w's neighborhood over w's label. In the following, we describe how we estimate this term.</Paragraph>
      <Paragraph position="32"> Neighborhood Features Each type of word relationship which constrains the assignment of SO labels to words (synonymy, antonymy, etc.) is mapped by OPINE to a neighborhood feature. This mapping allows OPINE to use simultaneously use multiple independent sources of constraints on the label of a particular word. In the following, we formalize this mapping. null Let T denote the type of a word relationship in R (synonym, antonym, etc.) and let Ak,T represent the labels assigned by Ak to neighbors of a word w which are connected to w through a relationship of type T . We have</Paragraph>
      <Paragraph position="34"> given Ak,T (see below). P(l(w) = L|uniontextT Ak,T)(m) is estimated combining the information from various features about w's label using the sigmoid function s():</Paragraph>
      <Paragraph position="36"> where c0,...cj are weights whose sum is 1 and which reflect OPINE 's confidence in each type of feature.</Paragraph>
      <Paragraph position="37"> Given word w, label L, relationship type T and neighborhood label assignment Ak, let NT represent the subset of w's neighbors connected to w through a type T relationship. The feature fT computes the probability that w's label is L given the labels assigned by Ak to words in NT . Using Bayes's Law and assuming that these labels are independent given l(w), we have the following formula for fT at iteration m:</Paragraph>
      <Paragraph position="39"> Lj if wj and w are linked by a relationship of type T and w has label L. We make the simplifying assumption that this probability is constant and depends only of T, L and Lprime, not of the particular words wj and w. For each tuple (T, L, Lj), L,Lj [?]{pos,neg,neutral}, OPINE builds a probability table using a small set of bootstrapped positive, negative and neutral words.</Paragraph>
      <Paragraph position="40"> 3.3.5 Finding (Word, Feature) SO Labels This subtask is motivated by the existence of frequent words which change their SO label based on associated features, but whose SO labels in the context of the respective features are consistent throughout the reviews (e.g., in the Hotel domain, &amp;quot;hot water&amp;quot; has a consistently positive connotation, whereas &amp;quot;hot room&amp;quot; has a negative one). In order to solve this task, OPINE first assigns each (w,f) pair an initial SO label which is w's SO label. The system then executes a relaxation labeling step during which syntactic relationships between words and, respectively, between features, are used to update the default SO labels whenever necessary. For example, (hot, room) appears in the proximity of (broken, fan). If &amp;quot;room&amp;quot;and &amp;quot;fan&amp;quot; are conjoined by and, this suggests that &amp;quot;hot&amp;quot; and &amp;quot;broken&amp;quot; have similar SO labels in the context of their respective features. If &amp;quot;broken&amp;quot; has a strongly negative semantic orientation, this fact contributes to OPINE's belief that &amp;quot;hot&amp;quot; may also be negative in this context. Since (hot, room) occurs in the vicinity of other such phrases (e.g., stifling kitchen), &amp;quot;hot&amp;quot; acquires a negative SO label in the context of &amp;quot;room&amp;quot;.</Paragraph>
      <Paragraph position="41"> 3.3.6 Finding (Word, Feature, Sentence) SO Labels This subtask is motivated by the existence of (w,f) pairs (e.g., (big, room)) for which w's orientation changes based on the sentence in which the pair appears (e.g., &amp;quot; I hated the big, drafty room because I ended up freezing.&amp;quot; vs. &amp;quot;We had a big, luxurious room&amp;quot;.) In order to solve this subtask, OPINE first assigns each (w,f,s) tuple an initial label which is simply the SO label for the (w,f) pair. The system then uses syntactic relationships between words and, respectively, features in order to update the SO labels when necessary. For example, in the sentence &amp;quot;I hated the big, drafty room because I ended up freezing.&amp;quot;, &amp;quot;big&amp;quot; and &amp;quot;hate&amp;quot; satisfy condition 2 in Table 5 and therefore OPINE expects them to have similar SO labels. Since &amp;quot;hate&amp;quot; has a strong negative connotation, &amp;quot;big&amp;quot; acquires a negative SO label in this context.</Paragraph>
      <Paragraph position="42"> In order to correctly update SO labels in this last step, OPINE takes into consideration the presence of negation modifiers. For example, in the sentence &amp;quot;I don't like a large scanner either&amp;quot;, OPINE first replaces the positive (w,f) pair (like, scanner) with the negative labeled pair (not like, scanner) and then infers that &amp;quot;large&amp;quot; is likely to have a negative SO label in this context.</Paragraph>
    </Section>
    <Section position="6" start_page="343" end_page="343" type="sub_section">
      <SectionTitle>
3.3.7 Identifying Opinion Phrases
</SectionTitle>
      <Paragraph position="0"> After OPINE has computed the most likely SO labels for the head words of each potential opinion phrase in the context of given features and sentences, OPINE can extract opinion phrases and establish their polarity. Phrases whose head words have been assigned positive or negative labels are retained as opinion phrases. Furthermore, the polarity of an opinion phrase o in the context of a feature f and sentence s is given by the SO label assigned to the tuple (head(o),f,s) (3.3.6 shows how OPINE takes into account negation modifiers).</Paragraph>
    </Section>
    <Section position="7" start_page="343" end_page="344" type="sub_section">
      <SectionTitle>
3.4 Experiments
</SectionTitle>
      <Paragraph position="0"> In this section we evaluate OPINE's performance on the following tasks: finding SO labels of words in the context of known features and sentences (SO label extraction); distinguishing between opinion and non-opinion phrases in the context of known features and sentences (opinion phrase extraction); finding the correct polarity of extracted opinion phrases in the context of known features and sentences (opinion phrase polarity extraction).</Paragraph>
      <Paragraph position="1"> While other systems, such as (Hu and Liu, 2004; Turney, 2002), have addressed these tasks to some degree, OPINE is the first to report results. We first ran OPINE on 13841 sentences and 538 previously extracted features.</Paragraph>
      <Paragraph position="2"> OPINE searched for a SO label assignment for 1756 different words in the context of the given features and sentences. We compared OPINE against two baseline methods, PMI++ and Hu++.</Paragraph>
      <Paragraph position="3"> PMI++ is an extended version of (Turney, 2002)'s method for finding the SO label of a phrase (as an attempt to deal with context-sensitive words). For a given (word, feature, sentence) tuple, PMI++ ignores the sentence, generates a phrase based on the word and the feature (e.g., (clean, room): &amp;quot;clean room&amp;quot;) and finds its SO label using PMI statistics. If unsure of the label, PMI++ tries to find the orientation of the potential opinion word instead. The search engine queries use domain-specific keywords (e.g., &amp;quot;scanner&amp;quot;), which are dropped if they  lead to low counts.</Paragraph>
      <Paragraph position="4"> Hu++ is a WordNet-based method for finding a word's context-independent semantic orientation. It extends Hu's adjective labeling method in a number of ways in order to handle nouns, verbs and adverbs in addition to adjectives and in order to improve coverage. Hu's method starts with two sets of positive and negative words and iteratively grows each one by including synonyms and antonyms from WordNet. The final sets are used to predict the orientation of an incoming word.</Paragraph>
      <Paragraph position="5">  in the Context of Given Product Features and Sentences.</Paragraph>
      <Paragraph position="6"> OPINE's precision is higher than that of PMI++ and Hu++.</Paragraph>
      <Paragraph position="7"> All results are reported with respect to PMI++ . Notation: adj=adjectives, nn=nouns, vb=verbs, adv=adverbs  On the task of finding SO labels for words in the context of given features and review sentences, OPINE obtains higher precision than both baseline methods at a small loss in recall with respect to PMI++. As described below, this result is due in large part to OPINE's ability to handle context-sensitive opinion words.</Paragraph>
      <Paragraph position="8"> We randomly selected 200 (word, feature, sentence) tuples for each word type (adjective, adverb, etc.) and obtained a test set containing 800 tuples. Two annotators assigned positive, negative and neutral labels to each tuple (the inter-annotator agreement was 78%). We retained the tuples on which the annotators agreed as the gold standard. We ran PMI++ and Hu++ on the test data and compared the results against OPINE's results on the same data.</Paragraph>
      <Paragraph position="9"> In order to quantify the benefits of each of the three steps of our method for finding SO labels, we also compared OPINE with a version which only finds SO labels for words and a version which finds SO labels for words in the context of given features, but doesn't take into account given sentences. We have learned from this comparison that OPINE's precision gain over PMI++ and Hu++ is mostly due to to its ability to handle context-sensitive words in a large number of cases.</Paragraph>
      <Paragraph position="10"> Although Hu++ does not handle context-sensitive SO label assignment, its average precision was reasonable (75%) and better than that of PMI++. Finding a word's SO label is good enough in the case of strongly positive or negative opinion words, which account for the majority of opinion instances. The method's loss in recall is due to not recognizing words absent from WordNet (e.g., &amp;quot;depth-adjustable&amp;quot;) or not having enough information to classify some words in WordNet.</Paragraph>
      <Paragraph position="11"> PMI++ typically does well in the presence of strongly positive or strongly negative words. Its high recall is correlated with decreased precision, but overall this simple approach does well. PMI++'s main shortcoming is misclassifying terms such as &amp;quot;basic&amp;quot; or &amp;quot;visible&amp;quot; which change orientation based on context.</Paragraph>
      <Paragraph position="12">  In order to evaluate OPINE on the tasks of opinion phrase extraction and opinion phrase polarity extraction in the context of known features and sentences, we used a set of 550 sentences containing previously extracted features. The sentences were annotated with the opinion phrases corresponding to the known features and with the opinion polarity. We compared OPINE with PMI++ and Hu++ on the tasks of interest. We found that OPINE had the highest precision on both tasks at a small loss in recall with respect to PMI++. OPINE's ability to identify a word's SO label in the context of a given feature and sentence allows the system to correctly extract opinions expressed by words such as &amp;quot;big&amp;quot; or &amp;quot;small&amp;quot;, whose semantic orientation varies based on context.</Paragraph>
      <Paragraph position="13">  OPINE's precision is higher than that of PMI++ and of Hu++. All results are reported with respect to PMI++.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>