<?xml version="1.0" standalone="yes"?>
<Paper uid="M92-1002">
  <Title>MUC-4 EVALUATION METRICS</Title>
  <Section position="2" start_page="0" end_page="0" type="intro">
    <SectionTitle>
INTRODUCTION
</SectionTitle>
    <Paragraph position="0"> The MUC-4 evaluation metrics measure the performance of the message understanding systems . This paper describes the scoring algorithms used to arrive at the metrics as well as the improvements that were made to th e MUC-3 methods. MUC-4 evaluation metrics were stricter than those used in MUC-3. Given the differences in scoring between MUC-3 and MUC-4, the MUC-4 systems' scores represent a larger improvement over MUC-3 performance than the numbers themselves suggest .</Paragraph>
    <Paragraph position="1"> The major improvements in the scoring of MUC-4 were the automation of the scoring of set fill slots, partia l automation of the scoring of string fill slots, content-based mapping enforced across the board, the focus on the AL L TEMPLATES score as opposed to the MATCHED/MISSING score in MUC-3, the exclusion of the template id scores from the score tallies, and the addition of the object level scores, string fills only scores, text filtering scores , and F-measures . These improvements and their effects on the scores are discussed in detail in this paper.</Paragraph>
  </Section>
class="xml-element"></Paper>