<?xml version="1.0" standalone="yes"?>
<Paper uid="P05-1045">
  <Title>Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling</Title>
  <Section position="7" start_page="367" end_page="368" type="evalu">
    <SectionTitle>
7 Results and Discussion
</SectionTitle>
    <Paragraph position="0"> In our experiments we compare the impact of adding the non-local models with Gibbs sampling to our baseline CRF implementation. In the CoNLL named entity recognition task, the non-local models increase the F1 accuracy by about 1.3%. Although such gains may appear modest, note that they are achieved relative to a near state-of-the-art NER system: the winner of the CoNLL English task reported an F1 score of 88.76. In contrast, the increases published by Bunescu and Mooney (2004) are relative to a baseline system which scores only 80.9% on the same task. Our performance is similar on the CMU Seminar Announcements dataset. We show the per-field F1 results that were reported by Sutton and McCallum (2004) for comparison, and note that we are again achieving gains against a more competitive baseline system.</Paragraph>
    <Paragraph position="1"> For all experiments involving Gibbs sampling, we used a linear cooling schedule. For the CoNLL dataset we collected 200 samples per trial, and for the CMU Seminar Announcements we collected 100 samples. We report the average of all trials, and in all cases we outperform the baseline with greater than 95% confidence, using the standard t-test. The trials had low standard deviations - 0.083% and 0.007% and high minimun F-scores - 86.72%, and 92.28%  - for the CoNLL and CMU Seminar Announcements respectively, demonstrating the stability of our method.</Paragraph>
    <Paragraph position="2"> The biggest drawback to our model is the computational cost. Taking 100 samples dramatically increases test time. Averaged over 3 runs on both Viterbi and Gibbs, CoNLL testing time increased from 55 to 1738 seconds, and CMU Seminar Announcements testing time increases from 189 to 6436 seconds.</Paragraph>
  </Section>
class="xml-element"></Paper>