<?xml version="1.0" standalone="yes"?>
<Paper uid="J00-2005">
  <Title>Squibs and Discussions Pipelines and Size Constraints</Title>
  <Section position="5" start_page="255" end_page="256" type="metho">
    <SectionTitle>
5. Discussion of Results
</SectionTitle>
    <Paragraph position="0"> As expected, the single-delta figures show that as the delta increases, both the average word count and the number of leaflets that exceed the size constraint also increase.</Paragraph>
    <Paragraph position="1"> Note that although none of the leaflets produced from the 150-questionnaire &amp;quot;tuning 3 Our revision module did not give any guidance as to where messages should be added. This sometimes led to wasted space in situations where a message could be added to one part of the leaflet but not others (for example, to the first inside page but not the second), if the next message in the undelete list was in a portion of the leaflet that had no unused space.</Paragraph>
    <Paragraph position="2"> 4 This measurement was made on a subset of 100 documents, because this is the size of collection that STOP was designed to be able to process in one run. While the core NLG system could process any number of documents, the support code (user-interface, logging, file management) worked poorly when processing more than 100-200 documents in one run. For word count and constraint violation data, we simply restarted the system if it hung when processing 1,000 questionnaires; but this seemed less appropriate for execution time data.</Paragraph>
    <Paragraph position="3">  set&amp;quot; violated the size constraint with a delta of 0, one leaflet produced from the full 2,582-questionnaire data set did break the size constraint at this delta. This is perhaps not surprising, it merely shows that as the size of the document set increases, so does the worst-case performance of the heuristic size estimator. It is possible that in a very large data set (hundreds of thousands of questionnaires), some leaflets might break the size constraint even at a delta of -1.</Paragraph>
    <Paragraph position="4"> Shifting to a multiple-solution pipeline dramatically improves performance. Average leaflet size while guaranteeing conformance to the size constraint jumps from 320 words in single-delta mode to 369 with two solutions; an increase of 15% in the number of words in the leaflet. We get still better results with three and four solutions, although the increase is not as dramatic. The best results of all are in revision mode, although the increase in size over a four-solution pipeline (385 words versus 380 words) is small. However, revision mode also is robust in the face of increased data set size (we can be confident that the size constraint will be satisfied even on a set of a million questionnaires) and &amp;quot;last-minute&amp;quot; changes to the code. If developers tweak the main STOP code and forget to update the size estimator, revision mode will still always produce documents that conform to the size constraint; it just may take longer to do the revision. In contrast, changes to the code may result in the multiple-solution pipeline producing documents that do not conform to the size constraint.</Paragraph>
    <Paragraph position="5"> As expected, processing time is lowest for the single-solution pipeline and highest for revision mode. However, in the context of STOP, even the 9.8 seconds required in revision mode is acceptable; under this mode a batch of 100 leaflets can still be generated in under 20 minutes.</Paragraph>
  </Section>
class="xml-element"></Paper>