<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-3241">
  <Title>The Entropy Rate Principle as a Predictor of Processing Effort: An Evaluation against Eye-tracking Data</Title>
  <Section position="2" start_page="0" end_page="0" type="intro">
    <SectionTitle>
1 Introduction
</SectionTitle>
    <Paragraph position="0"> Genzel and Charniak (2002, 2003) introduce the entropy rate principle, which states that speakers produce language whose entropy rate is on average constant. The motivation for this comes from information theory: the most efficient way of transmitting information through a noisy channel is at a constant rate. If human communication has evolved to be optimal in this sense, then we would expect humans to produce text and speech with approximately constant entropy. There is some evidence that this is true for speech (Aylett, 1999).</Paragraph>
    <Paragraph position="1"> For text, the entropy rate principle predicts that the entropy of an individual sentence increases with its position in the text, if entropy is measured out of context. Genzel and Charniak (2002) show that this prediction is true for the Wall Street Journal corpus, for both function words and for content words. They estimate entropy either using a language model or using a probabilistic parser; the effect can be observed in both cases. Genzel and Charniak (2003) extend this results in several ways: they show that the effect holds for different genres (but the effect size varies across genres), and also applies within paragraphs, not only within whole texts. Furthermore, they show that the effect can also be obtained for language other than English (Russian and Spanish). The entropy rate principle also predicts that a language model that takes context into account should yield lower entropy estimates compared to an out of context language model. Genzel and Charniak (2002) show that this prediction holds for caching language models such as the ones proposed by Kuhn and de Mori (1990).</Paragraph>
    <Paragraph position="2"> The aim of the present paper is to shed further light on the entropy rate effect discovered by Genzel and Charniak (2002, 2003) (henceforth G&amp;C) by providing new evidence in two areas.</Paragraph>
    <Paragraph position="3"> In Experiment 1, we replicate G&amp;C's entropy rate effect and investigate the source of the effect.</Paragraph>
    <Paragraph position="4"> The results show that the correlation coefficients that G&amp;C report are inflated by averaging over sentences with the same position, and by restricting the range of the sentence position considered. Once these restrictions are removed the effect is smaller, but still significant. We also show that the effect is to a large extend due to a confound with sentence length: longer sentences tend to occur later in the text. However, we are able to demonstrate that the entropy rate effect still holds once this confound has been removed.</Paragraph>
    <Paragraph position="5"> In Experiment 2, we test the psycholinguistic predictions of the entropy rate principle. This experiment uses a subset of the British National Corpus as training data and tests on the Embra corpus, a set of newspaper articles annotated with eye-movement data. We find that there is a correlation between the entropy of a sentence and the processing effort it causes, as measured by reading times in eye-tracking data. We also show that there is no correlation between processing effort and sentence position, which indicates that processing effort in context is constant through a text, which is one of the assumptions underlying the entropy rate principle.</Paragraph>
  </Section>
class="xml-element"></Paper>