<?xml version="1.0" standalone="yes"?>
<Paper uid="W06-1651">
  <Title>Joint Extraction of Entities and Relations for Opinion Recognition</Title>
  <Section position="5" start_page="432" end_page="433" type="metho">
    <SectionTitle>
3 Extraction of Opinion and Source
Entities
</SectionTitle>
    <Paragraph position="0"> We develop two separate sequence tagging classifiers for opinion extraction and source extraction, using linear-chain Conditional Random Fields (CRFs) (Lafferty et al., 2001). The sequence tagging is encoded as the typical 'BIO' scheme.4 Each training or test instance represents a sentence, encoded as a linear chain of tokens and their 4'B' is for the token that begins an entity, 'I' is for tokens that are inside an entity, and 'O' is for tokens outside an entity.</Paragraph>
    <Paragraph position="1">  associated features. Our feature set is based on that of Choi et al. (2005) for source extraction5, but we include additional lexical and WordNet-based features. For simplicity, we use the same features for opinion entity extraction and source extraction, and let the CRFs learn appropriate feature weights for each task.</Paragraph>
    <Section position="1" start_page="433" end_page="433" type="sub_section">
      <SectionTitle>
3.1 Entity extraction features
</SectionTitle>
      <Paragraph position="0"> For each token xi, we include the following features. For details, see Choi et al. (2005).</Paragraph>
      <Paragraph position="1"> word: words in a [-4, +4] window centered on xi.</Paragraph>
      <Paragraph position="2"> part-of-speech: POS tags in a [-2, +2] window.6 grammatical role: grammatical role (subject, object, prepositional phrase types) of xi derived from a dependency parse.7 dictionary: whether xi is in the opinion expression dictionary culled from the training data and augmented by approximately 500 opinion words from the MPQA Final Report8. Also computed for tokens in a [-1, +1] window and for xi's parent &amp;quot;chunk&amp;quot; in the dependency parse.</Paragraph>
      <Paragraph position="3"> semantic class: xi's semantic class.9 WordNet: the WordNet hypernym of xi.10</Paragraph>
    </Section>
  </Section>
  <Section position="6" start_page="433" end_page="434" type="metho">
    <SectionTitle>
4 Relation Classification
</SectionTitle>
    <Paragraph position="0"> We also develop a maximum entropy binary classifier for opinion-source link relation classification. Given an opinion-source pair, Oi-Sj, the relation classifier decides whether the pair exhibits a valid link relation, Li,j. The relation classifier focuses only on the syntactic structure and lexical properties between the two entities of a given pair, without knowing whether the proposed entities are correct. Opinion and source entities are taken from the n-best sequences of the entity extraction models; therefore, some are invariably incorrect.</Paragraph>
    <Paragraph position="1"> From each sentence, we create training and test instances for all possible opinion-source pairings that do not overlap: we create an instance for Li,j only if the span of Oi and Sj do not overlap.</Paragraph>
    <Paragraph position="2"> For training, we also filter out instances for which neither the proposed opinion nor source en- null tity overlaps with a correct opinion or source entity per the gold standard. This training instance filtering helps to avoid confusion between examples like the following (where entities marked in bold are the gold standard entities, and entities in square brackets represent the n-best output sequences from the entity extraction classifiers):  (1) [The president] s1 walked away from [the</Paragraph>
    <Paragraph position="4"> (2) [The monster] s2 walked away, [revealing] o4 a little box hidden underneath.</Paragraph>
    <Paragraph position="5"> For these sentences, we construct training instances for L1,1, L1,2, and L1,3, but not L2,4, which in fact has very similar sentential structure as L1,2, and hence could confuse the learning algorithm. null</Paragraph>
    <Section position="1" start_page="433" end_page="434" type="sub_section">
      <SectionTitle>
4.1 Relation extraction features
</SectionTitle>
      <Paragraph position="0"> The training and test instances for each (potential) link Li,j (with opinion candidate entity Oi and source candidate entity Sj) include the following features.</Paragraph>
      <Paragraph position="1"> opinion entity word: the words contained in Oi. phrase type: the syntactic category of the constituent in which the entity is embedded, e.g. NP or VP. We encode separate features for Oi and Sj. grammatical role: the grammatical role of the constituent in which the entity is embedded.</Paragraph>
      <Paragraph position="2"> Grammatical roles are derived from dependency parse trees, as done for the entity extraction classifiers. We encode separate features for Oi and Sj. position: a boolean value indicating whether Sj precedes Oi.</Paragraph>
      <Paragraph position="3"> distance: the distance between Oi and Sj in numbers of tokens. We use four coarse categories: adjacent, very near, near, far.</Paragraph>
      <Paragraph position="4"> dependency path: the path through the dependency tree from the head of Sj to the head of Oi. For instance, 'subj|verb' or 'subj|verb|obj'.</Paragraph>
      <Paragraph position="5"> voice: whether the voice of Oi is passive or active. syntactic frame: key intra-sentential relations between Oi and Sj. The syntactic frames that we use are: * [E1:role] [distance] [E2:role], where distance [?] {adjacent, very near, near, far}, and Ei:role is the grammatical role of Ei. Either E1 is an opinion entity and E2 is a source, or vice versa. * [E1:phrase] [distance] [E2:phrase], where Ei:phrase is the phrasal type of entity Ei.</Paragraph>
      <Paragraph position="6">  * [E1:phrase] [E2:headword], where E2 must be the opinion entity, and E1 must be the source entity (i.e. no lexicalized frames for sources). E1 and E2 can be contiguous.</Paragraph>
      <Paragraph position="7">  {by, of, from, for, between, among, and, have, be, will, not, ], &amp;quot;, ... }.</Paragraph>
      <Paragraph position="8"> When a syntactic frame is matched to a sentence, the bracketed items should be instantiated with particular values corresponding to the sentence. Pattern elements without square brackets are constants. For instance, the syntactic frame '[E1:phrase] NP [E2:phrase]' may be instantiated as 'VP NP VP'. Some frames are lexicalized with respect to the head of an opinion entity to reflect the fact that different verbs expect source entities in different argument positions (e.g. SOURCE blamed TARGET vs. TARGET angered SOURCE).</Paragraph>
    </Section>
  </Section>
  <Section position="7" start_page="434" end_page="435" type="metho">
    <SectionTitle>
5 Integer Linear Programming
Approach
</SectionTitle>
    <Paragraph position="0"> As noted in the introduction, we model our task as global, constraint-based inference over the separately trained entity and relation classifiers, and implement the inference procedure as binary integer linear programming (ILP) ((Roth and Yih, 2004), (Punyakanok et al., 2004)). ILP consists of an objective function which is a dot product between a vector of variables and a vector of weights, and a set of equality and inequality constraints among variables. Given an objective function and a set of constraints, LP finds the optimal assignment of values to variables, i.e. one that minimizes the objective function. In binary ILP, the assignments to variables must be either 0 or 1.</Paragraph>
    <Paragraph position="1"> The variables and constraints defined for the opinion recognition task are summarized in Table 1 and explained below.</Paragraph>
    <Paragraph position="2"> Entity variables and weights For each opinion entity, we add two variables, Oi and -Oi, where</Paragraph>
    <Paragraph position="4"> ensure coherent assignments, we add equality constraints [?]i, Oi + -Oi = 1. The weights woi and -woi for Oi and -Oi respectively, are computed as a negative conditional probability of the span of an entity to be extracted (or suppressed) given the labelings of the adjacent variables of the CRFs:</Paragraph>
    <Paragraph position="6"> where xm = 'O' for m [?] [k,l] where xi is the value assigned to the random variable of the CRF corresponding to an entity Oi. Likewise, for each source entity, we add two variables Sj and -Sj and a constraint Sj + -Sj = 1. The weights for source variables are computed in the same way as opinion entities.</Paragraph>
    <Paragraph position="7"> Relation variables and weights For each link relation, we add two variables Li,j and -Li,j, and a constraint Li,j + -Li,j = 1. By the definition of a link, if Li,j = 1, then it is implied that Oi = 1 and Sj = 1. That is, if a link is extracted, then the pair of entities for the link must be also extracted. Constraints to ensure this coherency are explained in the following subsection. The weights for link variables are based on probabilities from the binary link classifier.</Paragraph>
    <Paragraph position="8"> Constraints for link coherency In our corpus, a source entity can be linked to more than one opinion entity, but an opinion entity is linked to only  one source. Nonetheless, the majority of opinion-source pairs involve one-to-one mappings, which we encode as hard and soft constraints as follows: For each opinion entity, we add an equality constraint Oi = summationtextj Li,j to enforce that only one link can emanate from an opinion entity. For each source entity, we add an equality constraint and an inequality constraint that together allow a source to link to at most two opinions: Sj+Aj =summationtexti Li,j and Aj [?] Sj [?] 0, where Aj is an auxiliary variable, such that its weight is some positive constant value that suppresses Aj from being assigned to 1.</Paragraph>
    <Paragraph position="9"> And Aj can be assigned to 1 only if Sj is already assigned to 1. It is possible to add more auxiliary variables to allow more than two opinions to link to a source, but for our experiments two seemed to be a reasonable limit.</Paragraph>
    <Paragraph position="10"> Constraints for entity coherency When we use n-best sequences where n &gt; 1, proposed entities can overlap. Because this should not be the case in the final result, we add an equality constraint Xi +Xj = 1, X [?] {S,O} for all pairs of entities with overlapping spans.</Paragraph>
    <Paragraph position="11"> Adjustments to weights To balance the precision and recall, and to take into account the performance of different base classifiers, we apply adjustments to weights as follows.</Paragraph>
    <Paragraph position="12"> 1) We define six coefficients cx and -cx, where x [?] {O,S,L} to modify a group of weights as follows.</Paragraph>
    <Paragraph position="14"> In general, increasing cx will promote recall, while increasing -cx will promote precision.</Paragraph>
    <Paragraph position="15"> Also, setting co &gt; cs will put higher confidence on the opinion extraction classifier than the source extraction classifier.</Paragraph>
    <Paragraph position="16"> 2) We also define one constant cA to set the weights for auxiliary variable Ai. That is, [?]i, wAi := cA.</Paragraph>
    <Paragraph position="17"> 3) Finally, we adjust the confidence of the link variable based on n-th-best sequences of the entity extraction classifiers as follows.</Paragraph>
    <Paragraph position="18"> [?]i, wLi,j := wLi,j [?] d where d def= 4/(3 + min(m,n)), when Oi is from an m-th sequence and Sj is from a n-th sequence.11 11This will smoothly degrade the confidence of a link based on the entities from higher n-th sequences. Values of d decrease as 4/4, 4/5, 4/6, 4/7....</Paragraph>
  </Section>
  <Section position="8" start_page="435" end_page="436" type="metho">
    <SectionTitle>
6 Experiments-I
</SectionTitle>
    <Paragraph position="0"> We evaluate our system using the NRRC Multi-Perspective Question Answering (MPQA) corpus that contains 535 newswire articles that are manually annotated for opinion-related information.</Paragraph>
    <Paragraph position="1"> In particular, our gold standard opinion entities correspond to direct subjective expression annotations and subjective speech event annotations (i.e. speech events that introduce opinions) in the MPQA corpus (Wiebe et al., 2005). Gold standard source entities and link relations can be extracted from the agent attribute associated with each opinion entity. We use 135 documents as a development set and report 10-fold cross validation results on the remaining 400 documents in all experiments below.</Paragraph>
    <Paragraph position="2"> We evaluate entity and link extraction using both an overlap and exact matching scheme.12 Because the exact start and endpoints of the manual annotations are somewhat arbitrary, the overlap scheme is more reasonable for our task (Wiebe et al., 2005). We report results according to both matching schemes, but focus our discussion on results obtained using overlap matching.13 We use the Mallet14 implementation of CRFs.</Paragraph>
    <Paragraph position="3"> For brevity, we will refer to the opinion extraction classifier as CRF-OP, the source extraction classifier as CRF-SRC, and the link relation classifier as CRF-LINK. For ILP, we use Matlab, which produced the optimal assignment in a matter of few seconds for each sentence. The weight adjustment constants defined for ILP are based on the development data.15 The link-nearest baselines For baselines, we first consider a link-nearest heuristic: for each opinion entity extracted by CRF-OP, the link-nearest heuristic creates a link relation with the closest source entity extracted by CRF-SRC. Recall that CRF-SRC and CRF-OP extract entities from n-best sequences. We test the link-nearest heuristic with n = {1,2,10} where larger n will boost recall at the cost of precision. Results for the 12Given two links L1,1 = (O1,S1) and L2,2 = (O2,S2), exact matching requires the spans of O1 and O2, and the spans of S1 and S2, to match exactly, while overlap matching requires the spans to overlap.</Paragraph>
    <Paragraph position="4">  link-nearest heuristic on the full source-expressesopinion relation extraction task are shown in the first three rows of table 2. NEAREST-1 performs the best in overlap-match F-measure, reaching 59.9. NEAREST-10 has higher recall (66.3%), but the precision is really low (20.9%). Performance of the opinion and source entity classifiers will be discussed in Section 8.</Paragraph>
    <Paragraph position="5"> SRL baselines Next, we consider two baselines that use a state-of-the-art SRL system (Punyakanok et al., 2005). In many link relations, the opinion expression entity is a verb phrase and the source entity is in an agent argument position. Hence our second baseline, SRL, extracts all verb(V)-agent(A0) frames from the output of the SRL system and provides an upper bound on recall (59.7%) for systems that use SRL in isolation for our task. A more sophisticated baseline, SRL+CRF-OP, extracts only those V-A0 frames whose verb overlaps with entities extracted by the opinion expression extractor, CRF-OP. As shown in table 2, filtering out V-A0 frames that are incompatible with the opinion extractor boosts precision to 83.2%, but the F-measure (58.9) is lower than that of NEAREST-1.</Paragraph>
    <Paragraph position="6"> ILP results The ILP-n system in table 2 denotes the results of the ILP approach applied to the n-best sequences. ILP-10 reaches an F-measure of 68.0, a significant improvement over the highest performing baseline16, and also a substantial improvement over ILP-1. Note that the performance of NEAREST-10 was much worse than that 16Statistically significant by paired-t test, where p &lt; 0.001.</Paragraph>
    <Paragraph position="7">  ILP-n : ILP applied to n-best sequences ILP+SRL-f-n : ILP w/ SRL features, n-best ILP+SRL-fc-n : ILP w/ SRL features, and SRL constraints, n-best  of NEAREST-1, because the 10-best sequences include many incorrect entities whereas the corresponding ILP formulation can discard the bad entities by considering dependencies among entities and relations.17</Paragraph>
  </Section>
  <Section position="9" start_page="436" end_page="437" type="metho">
    <SectionTitle>
7 Additional SRL Incorporation
</SectionTitle>
    <Paragraph position="0"> We next explore two approaches for more directly incorporating SRL into our system.</Paragraph>
    <Paragraph position="1"> Extra SRL Features for the Link classifier We incorporate SRL into the link classifier by adding extra features based on SRL. We add boolean features to check whether the span of an SRL argument and an entity matches exactly. In addition, we include syntactic frame features as follows: * [E1:srl-arg] [E2:srl-arg], where Ei:srl-arg indicates the SRL argument type of entity Ei.</Paragraph>
    <Paragraph position="2"> * [E1.srl-arg] [E1:headword] [E2:srl-arg], where E1 must be an opinion entity, and E2 must be a source entity.</Paragraph>
    <Paragraph position="3"> Extra SRL Constraints for the ILP phase We also incorporate SRL into the ILP phase of our system by adding extra constraints based on SRL.</Paragraph>
    <Paragraph position="4"> In particular, we assign very high weights for links that match V-A0 frames generated by SRL, in order to force the extraction of V-A0 frames.</Paragraph>
    <Paragraph position="5"> 17A potential issue with overlap precision and recall is that the measures may drastically overestimate the system's performance as follows: a system predicting a single link relation whose source and opinion expression both overlap with every token of a document would achieve 100% overlap precision and recall. We can ensure this does not happen by measuring the average number of (source, opinion) pairs to which each correct or predicted pair is aligned (excluding pairs not aligned at all). In our data, this does not exceed 1.08, (except for baselines), so we can conclude these evaluation measures are behaving reasonably.</Paragraph>
  </Section>
class="xml-element"></Paper>