<?xml version="1.0" standalone="yes"?>
<Paper uid="W06-1651">
  <Title>Joint Extraction of Entities and Relations for Opinion Recognition</Title>
  <Section position="10" start_page="437" end_page="438" type="concl">
    <SectionTitle>
8 Experiments-II
</SectionTitle>
    <Paragraph position="0"> Results using SRL are shown in Table 3 (on the previous page). In the table, ILP+SRL-f denotes the ILP approach using the link classifier with the extra SRL 'f'eatures, and ILP+SRL-fc denotes the ILP approach using both the extra SRL 'f'eatures and the SRL 'c'onstraints. For comparison, the ILP-1 and ILP-10 results from Table 2 are shown in rows 1 and 2.</Paragraph>
    <Paragraph position="1"> The F-measure score of ILP+SRL-f-10 is 68.9, about a 1 point increase from that of ILP-10, which shows that extra SRL features for the link classifier further improve the performance over our previous best results.18 ILP+SRL-fc-10 also performs better than ILP-10 in F-measure, although it is slightly worse than ILP+SRL-f-10.</Paragraph>
    <Paragraph position="2"> This indicates that the link classifier with extra SRL features already makes good use of the V-A0 frames from the SRL system, so that forcing the extraction of such frames via extra ILP constraints only hurts performance by not allowing the extraction of non-V-A0 pairs in the neighborhood that could have been better choices.</Paragraph>
    <Paragraph position="3"> Contribution of the ILP phase In order to highlight the contribution of the ILP phase for our task, we present 'before' and 'after' performance in Table 4. The first row shows the performance of the individual CRF-OP, CRF-SRC, and CRF-LINK classifiers before the ILP phase. Without the ILP phase, the 1-best sequence generates the best scores. However, we also present the performance with merged 10-best entity sequences19 in order to demonstrate that using 10-best sequences without ILP will only hurt performance. The precision of the merged 10-best sequences system is very low, however the recall level is above 95% for both 18Statistically significant by paired-t test, where p &lt; 0.001.</Paragraph>
    <Paragraph position="4"> 19If an entity Ei extracted by the ith-best sequence overlaps with an entity Ej extracted by the jth-best sequence, where i &lt; j, then we discard Ej. If Ei and Ej do not overlap, then we extract both entities.</Paragraph>
    <Paragraph position="5"> CRF-OP and CRF-SRC, giving an upper bound for recall for our approach. The third row presents results after the ILP phase is applied for the 10-best sequences, and we see that, in addition to the improved link extraction described in Section 7, the performance on source extraction is substantially improved, from F-measure of 73.9 to 78.1. Performance on opinion expression extraction decreases from F-measure of 81.9 to 78.8. This decrease is largely due to implicit links, which we will explain below. The fourth row takes the union of the entities from ILP-SRL-f-10 and the entities from the best sequences from CRF-OP and CRF-SRC. This process brings the F-measure of CRF-OP up to 82.0, with a different precision-recall break down from those of 1-best sequences without ILP phase. In particular, the recall on opinion expressions now reaches 82.3%, while maintaining a high precision of 81.7%.</Paragraph>
    <Paragraph position="6">  justment. (All cases using ILP+SRL-f-10) Effects of ILP weight adjustment Finally, we show the effect of weight adjustment in the ILP formulation in Table 5. The DEV.CONF row shows relation extraction performance using a weight configuration based from the development data.</Paragraph>
    <Paragraph position="7"> In order to see the effect of weight adjustment, we ran an experiment, NO.CONF, using fixed default weights.20 Not surprisingly, our weight adjustment tuned from the development set is not the optimal choice for cross-validation set. Nevertheless, the weight adjustment helps to balance the precision and recall, i.e. it improves recall at the 20To be precise, cx = 1.0,-cx = 1.0 for x [?] {O,S,L}, but cA = 0.2 is the same as before.</Paragraph>
    <Paragraph position="8">  cost of precision. The weight adjustment is more effective when the gap between precision and recall is large, as was the case with the development data.</Paragraph>
    <Paragraph position="9"> Implicit links A good portion of errors stem from the implicit link relation, which our system did not model directly. An implicit link relation holds for an opinion entity without an associated source entity. In this case, the opinion entity is linked to an implicit source. Consider the following example.</Paragraph>
    <Paragraph position="10"> * Anti-Soviet hysteria was firmly oppressed.</Paragraph>
    <Paragraph position="11"> Notice that opinion expressions such as &amp;quot;Anti-Soviet hysteria&amp;quot; and &amp;quot;firmly oppressed&amp;quot; do not have associated source entities, because sources of these opinion expressions are not explicitly mentioned in the text. Because our system forces each opinion to be linked with an explicit source entity, opinion expressions that do not have explicit source entities will be dropped during the global inference phase of our system. Implicit links amount to 7% of the link relations in our corpus, so the upper bound for recall for our ILP system is 93%. In the future we will extend our system to handle implicit links as well. Note that we report results against a gold standard that includes implicit links. Excluding them from the gold standard, the performance of our final system ILP+SRL-f-10 is 72.6% in recall, 72.4% in precision, and 72.5 in F-measure.</Paragraph>
  </Section>
class="xml-element"></Paper>