<?xml version="1.0" standalone="yes"?>
<Paper uid="P06-1052">
  <Title>Sydney, July 2006. c(c)2006 Association for Computational Linguistics An Improved Redundancy Elimination Algorithm for Underspecified Representations</Title>
  <Section position="7" start_page="413" end_page="415" type="evalu">
    <SectionTitle>
5 Evaluation
</SectionTitle>
    <Paragraph position="0"> In this final section, we evaluate the the effectiveness and efficiency of the elimination algorithm: We run it on USRs from a treebank and measure how many readings are redundant, to what extent the algorithm eliminates this redundancy, and how much time it takes to do this.</Paragraph>
    <Paragraph position="1"> Resources. The experiments are based on the Rondane corpus, a Redwoods (Oepen et al., 2002) style corpus which is distributed with the English Resource Grammar (Flickinger, 2002). The corpus contains analyses for 1076 sentences from the tourism domain, which are associated with USRs based upon Minimal Recursion Semantics (MRS).</Paragraph>
    <Paragraph position="2"> The MRS representations are translated into dominance graphs using the open-source utool tool (Koller and Thater, 2005a), which is restricted to MRS representations whose translations are hnc.</Paragraph>
    <Paragraph position="3"> By restricting ourselves to such MRSs, we end up with a data set of 999 dominance graphs. The average number of scope bearing operators in the data setis6.5,andthemediannumberofreadingsis56.</Paragraph>
    <Paragraph position="4"> We then defined a (rather conservative) rewrite system RERG for capturing the permutability relation of the quantifiers in the ERG. This amounted to 34 rule schemata, which are automatically expanded to 494 rewrite rules.</Paragraph>
    <Paragraph position="5"> Experiment: Reduction. We first analysed the extent to which our algorithm eliminated the redundancy of the USRs in the corpus. We computed dominance charts for all USRs, ran the algorithm on them, and counted the number of configurations of the reduced charts. We then compared these numbers against a baseline and an upper bound. The upper bound is the true number of  equivalence classes with respect to RERG; for efficiency reasons we could only compute this number for USRs with up to 500.000 configurations (95% of the data set). The baseline is given by the number of readings that remain if we replace proper names and pronouns by constants and variables, respectively. This simple heuristic is easy to compute, and still achieves nontrivial redundancy elimination because proper names and pronouns are quite frequent (28% of the noun phrase occurrences in the data set). It also shows the degree of non-trivial scope ambiguity in the corpus.</Paragraph>
    <Paragraph position="6"> For each measurement, we sorted the USRs according to the number N of configurations, and grouped USRs according to the natural logarithm of N (rounded down) to obtain a logarithmic scale.</Paragraph>
    <Paragraph position="7"> First, we measured the mean reduction factor for each log(N) class, i.e. the ratio of the number of all configurations to the number of remaining configurations after redundancy elimination (Fig. 5). The upper-bound line in the figure shows thatthereisagreatdealofredundancyintheUSRs in the data set. The average performance of our algorithm is close to the upper bound and much  better than the baseline. For USRs with fewer than e8 =2980configurations(83%ofthedataset),the mean reduction factor of our algorithm is above 86% of the upper bound. The median number of configurations for the USRs in the whole data set is 56, and the median number of equivalence classes is 3; again, the median number of configurations of the reduced charts is very close to the upper bound, at 4 (baseline: 8). The highest reduction factor for an individual USR is 666.240.</Paragraph>
    <Paragraph position="8"> We also measured the ratio of USRs for which thealgorithmachievescompletereduction(Fig.6): The algorithm is complete for 56% of the USRs in the data set. It is complete for 78% of the USRs with fewer than e5 = 148 configurations (64% of the data set), and still complete for 66% of the USRs with fewer than e8 configurations.</Paragraph>
    <Paragraph position="9"> Experiment: Efficiency. Finally, we measured the runtime of the elimination algorithm. The run-time of the elimination algorithm is generally comparable to the runtime for computing the chart in the first place. However, in our experiments we used an optimised version of the elimination algorithm, which computes the reduced chart directly from a dominance graph by checking each split for eliminability before it is added to the chart.</Paragraph>
    <Paragraph position="10"> We compare the performance of this algorithm to the baseline of computing the complete chart. For comparison, we have also added the time it takes to enumerate all configurations of the graph, as a lower bound for any algorithm that computes the equivalence classes based on the full set of configurations. Fig. 7 shows the mean runtimes for each log(N) class, on the USRs with less than one million configurations (958 USRs).</Paragraph>
    <Paragraph position="11"> As the figure shows, the asymptotic runtimes for computing the complete chart and the reduced chart are about the same, whereas the time for  enumerating all configurations grows much faster.</Paragraph>
    <Paragraph position="12"> (Note that the runtime is reported on a logarithmic scale.) For USRs with many configurations, computing the reduced chart actually takes less time on average than computing the complete chart because the chart-filling algorithm is called on fewer subgraphs. While the reduced-chart algorithm seems to be slower than the complete-chart one for USRs with less than e5 configurations, these runtimes remain below 20 milliseconds on average, and the measurements are thus quite unreliable. In summary, we can say that there is no overhead for redundancy elimination in practice.</Paragraph>
  </Section>
class="xml-element"></Paper>