<?xml version="1.0" standalone="yes"?>
<Paper uid="C04-1008">
  <Title>Annotating and measuring temporal relations in texts</Title>
  <Section position="1" start_page="0" end_page="0" type="abstr">
    <SectionTitle>
Abstract
</SectionTitle>
    <Paragraph position="0"> This paper focuses on the automated processing of temporal information in written texts, more specifically on relations between events introduced by verbs in finite clauses. While this problem has been largely studied from a theoretical point of view, it has very rarely been applied to real texts, if ever, with quantified results. The methodology required is still to be defined, even though there have been proposals in the strictly human annotation case. We propose here both a procedure to achieve this task and a way of measuring the results. We have been testing the feasibility of this on newswire articles, with promising results.</Paragraph>
    <Paragraph position="1"> 1 Annotating temporal information This paper focuses on the automated annotation of temporal information in texts, more specifically relations between events introduced by finite verbs. While the semantics of temporal markers and the temporal structure of discourse are well-developed subjects in formal linguistics (Steedman, 1997), investigation of quantifiable annotation of unrestricted texts is a somewhat recent topic. The issue has started to generate some interest in computational linguistics (Harper et al., 2001), as it is potentially an important component in information extraction or question-answer systems. A few tasks can be distinguished in that respect: detecting dates and temporal markers detecting event descriptions finding the date of events described figuring out the temporal relations between events in a text The first task is not too difficult when looking for dates, e.g. using regular expressions (Wilson et al., 2001), but requires some syntactic analysis in a larger framework (Vazov, 2001; Shilder and Habel, 2001). The second one raises more difficult, ontological questions; what counts as an event is not uncontroversial (Setzer, 2001): attitude reports, such as beliefs, or reported speech have an unclear status in that respect. The third task adds another level of complexity: a lot of events described in a text do not have an explicit temporal stamp, and it is not always possible to determine one, even when taking context into account (Filatova and Hovy, 2001). This leads to an approach more suited to the level of underspecification found in texts: annotating relations between events in a symbolic way (e.g. that an event e1 is before another one e2). This is the path chosen by (Katz and Arosio, 2001; Setzer, 2001) with human annotators. This, in turn, raises new problems. First, what are the relations best suited to that task, among the many propositions (linguistic or logical) one can find for expressing temporal location ? Then, how can an annotation be evaluated, between annotators, or between a human annotator and an automated system ? Such annotations cannot be easy to determine automatically anyway, and must use some level of discourse modeling (cf. the work of (Grover et al., 1995)).</Paragraph>
    <Paragraph position="2"> We want to show here the feasibility of such an effort, and we propose a way of evaluating the success or failure of the task. The next section will precise why evaluating this particular task is not a trivial question. Section 3 will explain the method used to extract temporal relations, using also a form of symbolic inference on available temporal information (section 4). Then section 5 discusses how we propose to evaluate the success of the task, before presenting our results (section 6).</Paragraph>
  </Section>
class="xml-element"></Paper>